Questions about defining fields within schemas...

This is the forum for Murmurations.network

Questions about defining fields within schemas...

wu-lee

Rules for naming fields

Are there any reserved (or special) fields which schemas either shouldn't, or must, define? (e.g. 'linked_schemas' field, which I ask about specifically here: https://murmurations.flarum.cloud/d/21-the-linked-schemas-field)
Are there any rules or guidelines for naming fields? What characters can they contain? Should they be snake-case, camelCase or something else?

Field encoding

Mixed content for text is a problem in various datasets I've used, and generally results having to code defensively and/or convert everything to a lowest common denominator, like plain text. But this can make headings, bullet-points and paragraphs collapse resulting a very obtusely-worded result.

Encoding mismatches can't easily be detected and rectified, resulting in what's called "Mojibake" when displayed in user interfaces (web pages etc.):

https://en.wikipedia.org/wiki/Mojibake

Should I assume any default encoding for text fields? (e.g. Latin1, UTF-8 etc.) I'd like to be able to define this for my fields, or at least signal what this should be to authors.
Likewise, for formatting within text fields like description? (e.g. HTML, Markdown, etc.)

I'd suggest Murmurations should aim to specify encoding and possibly allowed formatting in the library fields. Fields like description obviously benefit from this, but others will do too: for instance standard URLs cannot contain Unicode glyphs unless URL-encoded with percent characters, but the URLs ("IRI"s) used in linked-data can.

Extending fields

Can constraints be added on existing field definitions?

For instance, the url-v1.json field schema's regular-expression validation pattern has no anchor on the URL scheme (https?://), and is extremely permissive about what follows. Presumably deliberate, but I wonder what scope there is for authors to tighten this up? Not to mention that defining an URL-validating regular expression is not especially trivial and requires rather technical know-how. Email fields are even harder...

Also some field definitions might almost but not quite map from existing fields in other datasets (like ours) when the definition is subtly different to the Murmuration field library's. For instance, countries and addresses are hard to map onto the locality field when the address has different components. Countries in particular may not correspond to an ISO 3166-1 two-letter code, and even if they do, the ISO definition changes slowly as countries are declared or redefined.

https://en.wikipedia.org/wiki/ISO_3166-1#Changes

Our ESSGLOBAL vocabularies are hard to map - for now I'm just converting the English labels for "Organisational Structure" terms into org_type_tags.

geoffturk

wu-lee Are there any reserved (or special) fields which schemas either shouldn't, or must, define? (e.g. 'linked_schemas' field, which I ask about specifically here: https://murmurations.flarum.cloud/d/21-the-linked-schemas-field)

Yes, linked_schemas is the only field that is required in all schemas, in order to ensure a profile always has at least one schema against which it must validate.

There are some other fields that if provided will be recorded in the Murmurations Index.

Are there any rules or guidelines for naming fields? What characters can they contain? Should they be snake-case, camelCase or something else?

See this GitHub discussion for our working guidelines on naming conventions. In summary, lowercase alphanumeric characters with underscore separators (first_name) is the standard for field names.

olisb

Hiya - there's a bunch of detail here - and I'm struggling to keep up!
But, briefly:

For encoding, Murmurations uses UTF-8 since it handles international character sets really well.

Murmurations doesn't really accommodate formatting (HTML, Markdown, etc.) though it could be included. Aggregators may or may not display it correctly though (i.e., they might just show <b>MyOrg</b> instead of a bolded MyOrg).

wu-lee

olisb Thanks for taking the time to understand and answer my questions.

I'll attempt to answer some of them myself, perhaps it's easier to see where I'm coming from.

Field naming: I infer from your answer in another thread that yes, linked_schemas is a reserved field. In that, it must exist and be defined to use the same (or a compatible) definition to the other schemas, for the index validation to work normally. What isn't clear what would happen if I used the name with a different field definition, or used the linked_schemas definition for a different name? But I guess it's safe to say: "don't do that or your profile may be rejected and not indexed".

What other special fields exist? Well, this isn't explicit, but I don't think there the aggregator could create a map without coordinates, so something must be looking for a field of a certain name and/or type which can be used to place a map pin. Likewise, I think something must be looking for a name to put on that pin...? I am not sure what to do here except to try and blindly copy the name and geolocation fields in the murmurations-map schema.

Aside: this does make me wonder if I should in fact be aiming to first mix in existing schemas with linked_schemas and then only adding new fields where there are semantic gaps to fill, rather than starting afresh and possibly making something which doesn't play nicely. linked_schemas then becomes a way to add in components like "mappable things", "web-based things", "addressable things" etc.

Allowed characters: The default-v1 schema includes a declaration which seems to limit field names to lowercase-alphanumeric-plus-underscore characters:

 "propertyNames": {
    "pattern": "^[a-z0-9_]*$"
  },

However, the others don't include that, so it seems to be entirely up to the schema author whether to restrict this. In general, there seems not to be any restriction.

I think it would be most robust if we can add a supplementary schema to the JSON schema which pins things like this down. (I don't currently know for sure this is possible, but it seems feasible.) But at least perhaps this can be added to the authors' guide.

olisb

wu-lee What other special fields exist?

none

linked_schemas only works within a profile, and is at the moment the only required field from the POV of the Index.

Well, this isn't explicit, but I don't think there the aggregator could create a map without coordinates, so something must be looking for a field of a certain name and/or type which can be used to place a map pin.

Yeah - see my answer on your other thread:

The demo (or any aggregator, really) can only plot points on a map if it has geolocation and 'something-to-plot' hence why the test_schema uses the fields name-v1 and geolcation-v1.

But Murmurations schemas don't have to include this info, because an aggregator may not want to make a map - so they may not need name or geolocation.

Likewise, I think something must be looking for a name to put on that pin...? I am not sure what to do here except to try and blindly copy the name and geolocation fields in the murmurations-map schema.

It's really up to you to use the existing fields you need from the Field Lib or, to create any new fields you need for your specific purposes and, ideally, to specify their context clearly so that other schema creators can re-use your field so that more data becomes interoperable.

wu-lee The default-v1 schema includes a declaration which seems to limit field names to lowercase-alphanumeric-plus-underscore characters:

All property names (i.e. field names) must follow the pattern e.g.

"field": { "name": "image-v1", "version": 1 },

This should probably be part of the docs too - yes!

wu-lee

Ok, this helps.

I'll now think out loud here: you don't need to answer these right now @olisb unless you have time and the inclination.

I'm starting to think that all this implies schemas in the library must not share field names... except when they mean exactly the same thing (like linked_schemas must). The name implies the meaning, and the fields must have compatible (identical) definitions. Correct?

So although I can define a field referencing the geolocation-v1.json field definition in my_schema and call it my_geolocation, profiles using my_schema as well as the murmurations_map-v1 schema would then have up to two fields defining a geographic coordinate - but these would be distinct and unrelated fields. If such a profile included a my_geolocation field, but no geolocation field, presumably it would not show up on the murmurations map aggregator, which identifies the geolocation field by the name in the murmuration_map-v1 schema. But it will always ignore my_geolocation. Correct?

If "the fields must have compatible (identical) definitions" is true, then I think this means that I could declare a field called geolocation in my schema, but I could not define it to use a different semantic interpretation of the coordinates (e.g. use different units or origin), nor a different type or syntax (e.g. {"lat": 0, "lon": 0} vs {"lat": 0, "long": 0} vs [0, 0] vs "0,0").

Maybe I could use a definition which is merely compatible (matches a subset of values), or equivalent (matches the same set), or maybe I'd have to define it in literally an identical way, as "$ref":"../fields/geolocation-v1.json" (because that's easier to check)?

I think this would make redefining the geolocation field to use an incompatible (non-subset) definition of geolocation-v2.json impossible in any schema unless it is done in every schema?

olisb

wu-lee So although I can define a field referencing the geolocation-v1.json field definition in my_schema and call it my_geolocation, profiles using my_schema as well as the murmurations_map-v1 schema would then have up to two fields defining a geographic coordinate - but these would be distinct and unrelated fields.

No - if you reference the existing field: geolocation-v1.json in my_schema, it is THE SAME field which the murmurations_map-v1 schema uses, so there will only be ONE set of geolocation coordinaites in the profile file.

Have a play with the MPG, select just the Test schema, make a profile, then go back and add the Murmurations schema... Notice how the form DOES NOT add new "name" and "geolocation" fields?? The MPG knows you are making just one profile and that the info in that profile is used across TWO schemas.

If it didn't work like this there would not be much point in Murmurations because none of the data would be interoperable!

Sorry, I really thought you had already understood this, it's a key part of Murmurations.
We obviously need to make it more clear!

Oli

wu-lee

Now, thinking through my solidarity_economy_initiatives-v0.1.0 schema from this point of view, it seems most of it is already replicating the murmurations_map fields, sometimes with the same field names and sometimes with different field names.

solidarity_economy_initiatives-v0.10 fields are:

"ld_uri": { "$ref": "../fields/url-v1.json" }, /** required **/
"name": { "$ref": "../fields/name-v1.json" }, /** required **/
"description": { "$ref": "../fields/description-v1.json" },
"website": { "$ref": "../fields/url-v1.json" },
"location": { "$ref": "../fields/location-v1.json" },
"geolocation": { "$ref": "../fields/geolocation-v1.json" },
"org_type_tags": { "$ref": "../fields/org_type_tags-v1.json" }

And murmurations_map-v1 fields are:

"linked_schemas": { "$ref": "../fields/linked_schemas-v1.json" }, /** required **/
"name": { "$ref": "../fields/name-v1.json" }, /** required **/
"urls": { "$ref": "../fields/urls-v1.json" }, /** required **/
"description": { "$ref": "../fields/description-v1.json" },
"mission": { "$ref": "../fields/mission-v1.json" },
"location": { "$ref": "../fields/location-v1.json" },
"geolocation": { "$ref": "../fields/geolocation-v1.json" },
"image": { "$ref": "../fields/image-v1.json" },
"rss": { "$ref": "../fields/rss-v1.json" },
"org_type_tags": { "$ref": "../fields/org_type_tags-v1.json" }

So if a profile wanted to be on the SEA map and the Murmurations map, it would have to name both in linked_schemas (if only because of the latter schema's requirement), but get by with just one name, uris and geolocation field.

However, if I wanted to say what the website was on the SEA map, I would also need the website field, even if this duplicates the urls field.

The org_type_tags, location and description fields are optional, and recognised by both map aggregators.

The solidarity_economy_initiatives-v0.10 schema's ld_uri would not be interpreted as a website URL by either, despite the field definition being url-v1, because the name is distinct.

I wonder if it's sensible to redefining any of the fields in solidarity_economy_initiatives-v0.10 which are duplicates of those in murmurations_map-v1? Specifically: name, description, location, geolocation, and org_type_tags. I'm thinking not: I'd be better adding the murmurations_map-v1 as a linked schema in those profiles which need it, only define new and distinct fields in solidarity_economy_initiatives-v0.10, and only do that when there are no existing schemas to sensibly link to provide them.

In this case, that seems to slim my schema down to just the ld_uri field and a linked_schema field. The other fields can still be present in the profiles as before, so long as the murmurations_map-v1 schema is listed in linked_schema.

Possible snags with this could be:

I want there to be just one URL, not many, and I want it to be mandatory.
My name, description and location fields are (perhaps!) in Gaelic
description contains HTML markup with some unusual unicode glyphs; and
The org_type_tags are my own categories, using alphanumeric identifiers like "A20"

(These examples are partly based on reality, and partly made up to make a point. In fact I am currently using English terms because the context seems to want it. But some descriptions are not English, some include HTML, and category identifiers are more machine-friendly and canonical, because this avoids language and spelling variance.)

Mixing our profile data with existing profile data could make for a dissonant experience for users of the Murmurations Map Aggregator, because they are built with different assumptions about these fields. Therefore, perhaps it is sensible to define new fields for these in solidarity_economy_initiatives-v0.10, even if they do duplicate information in the murmurations_map-v1. I could then add logic to translate terms from Gaelic/identifiers to English (if that is expected) for the Murmurations Map, and to duplicate the website field into the urls field, plus any other mapping required.

wu-lee

Thanks for your patience with my questions so far.

olisb No - if you reference the existing field: geolocation-v1.json in my_schema, it is THE SAME field which the murmurations_map-v1

I do understand that if two schemas like murmurations_map-v1 and test_schema-v1 define two identically named fields with the same field schema, these will be merged. There's not much room for ambiguity there.

I am wondering about the less obvious cases, when the field names differ, or the field types differ?

Take one field: geolocation. First, if the second schema adds a new geolocation-v1.json field called my_geolocation, I think this will not be merged in profiles. So potentially there will be two geolocation-v1.json fields in the profile with different names.

I think that applications like the Map Aggregator will only identify fields in profiles by their names - so it will ignore the second field my_geolocation, even if the first is absent - except perhaps as an opaque "additional field" in the profile.

Secondly, perhaps my schema used the same name geolocation but a different type:

"geolocation": { "$ref": "../fields/geolocation-v2.json" }

When the types are wildly different, I would expect something - perhaps human eyeballs, or a merge validation rule - to reject this as invalid, because the fields then cannot be merged meaningfully.

Therefore, it appears that any fields with the same name must have identical definitions, in whatever schema they appear, to avoid contradicting each other when a profiles lists them both. (Unless something clever is done to merge "compatible" schema, to support field versioning, which seems difficult.)

These points seem important to state, and were not obvious to me earlier on - perhaps I didn't read the right things?

To sum up, I don't have a good conclusion to this. I'm still uncertain how to design my schema: if fields of the type I need already exist in a schema in the library - like name or geolocation above - is there any sense to include similar ones in a new schema? Seems not, if the meaning is the same - but the schema examples currently in MurmurationsLibrary seem to do exactly that. So are these situations when this can be justified? Perhaps the meanings are different? Or perhaps the schema examples ought to be trimmed to eliminate field duplication?

In this context, the solidarity-economy-initiative schema as it is seems to add very little utility: perhaps the one ld_uri field which links it back to the original descriptions has a genuinely distinct meaning. Perhaps instead I should generalise this field into a schema for any profile which needs a linked data URI distinct from its URL? And then focus on new schema containing fields which are conceptually distinct from any other schema's but related to each other - for example, using terms to describe organisations using the ESSGLOBAL vocabulary.

geoffturk

wu-lee Therefore, it appears that any fields with the same name must have identical definitions, in whatever schema they appear, to avoid contradicting each other when a profiles lists them both. (Unless something clever is done to merge "compatible" schema, to support field versioning, which seems difficult.)

These points seem important to state, and were not obvious to me earlier on - perhaps I didn't read the right things?

This is a good point. I will add it to the docs site:
https://docs.murmurations.network/guides/create-a-schema.html

To sum up, I don't have a good conclusion to this. I'm still uncertain how to design my schema: if fields of the type I need already exist in a schema in the library - like name or geolocation above - is there any sense to include similar ones in a new schema? Seems not, if the meaning is the same - but the schema examples currently in MurmurationsLibrary seem to do exactly that. So are these situations when this can be justified? Perhaps the meanings are different? Or perhaps the schema examples ought to be trimmed to eliminate field duplication?

Fields need to carry a certain meaning. Aside from a label and description, we have started to add a purpose field inside a field's metadata where there can be a long-form explanation of the semantic meaning of that field. A context field can also be added which should be an array of strings with URLs that link to definitions (e.g., a Schema.org definition).

There are different ways to approach schema writing. One is to create a bunch of modular components and then piece them together. The classic example being create some sort of "base" schema and then add "extensions" on to it. Another is to link to a few field definition and then add in custom ones directly into the schema. We don't really know how schema creators will want to use Murmurations, so we have tried to keep open as many options as possible so that nobody is constrained or boxed into an approach that may not work well for them.

olisb

I think we may make more progress via a chat.
Would you be up for a call on BBB at 2pm UK time on Friday?

wu-lee

olisb Sure.

This is the forum for Murmurations.network