Common Definitions Simplification

[egekorkan]

Description of Changes

I am taking the proposal in #2165 and adapting to it. The best is to look at the diff in the tds.js file. Here are my takeaways:

• Amount of possible examples (the first test cases) has drastically reduced. That means, there is less ways to express the same concept. This is a big win.
• Simple examples have basically no impact. I only had to replace form with formDefaults.
• When all affordances have multiple protocols but the same security or same contentType, the TD gets longer. From a "mechanical" perspective this is negative.
• Extending the above point, longer TDs get easier to understand from a human perspective. They can be better described I would say. That means an implementation who can only support this case and above is easier to implement.
• When all affordances and forms DO NOT use the same form definition (i.e., events with mqtt, writeproperty requiring basic instead of nosec, etc.) the TDs are still weird. They were weird before anyways. An implementation that understands this is going to be more difficult to implement. These are less common but I would not call them edge cases.

As a result, I like this approach more.

Related Issue

Closes #2165

Type of Change

• Editorial (non-normative) with label .

[egekorkan]

Some discussion points:

• having formDefaults in the forms or not -> not having this eliminates some use cases, requiring expanded TDs
• using inherit can eliminate the copy pasting of the point 3 above.

[egekorkan]

TD call today:

• Having a term like form in the forms instead of formDefaults is probably better. It is not default anymore, it is usage. It makes it clearer to understand. -> @egekorkan will do the change
• The previous example with "matrix multiplication" gets simplified. The change in example 11 (

  wot-thing-description/proposals/common-definitions/tooling/tds.js

  Line 728 in e1bb3d4

  // 11. multiple content types and protocols where all affordances are available in json and cbor AND coap and http

  ) is appreciated by the TF.

[egekorkan]

I have implemented the feedback and this is aligned more with @benfrancis proposal. Some thoughts:

• I am not sure how we can do combo security with the current mechanism. I think going towards how open API defines it might make more sense (object for OR, array for AND). Even that is not possible here. As combo security is not the most common security configuration, I would be ok to make it ugly.
• base is usable in a definition but not in the form level. form is vice versa. Makes validation a bit annoying.
• If this direction is nice for everyone and my tooling PR at Eclipse Thingweb goes nicely, I would like to move this to the ED. @mahdanoura that will imply breaking changes that you need to catch up in the tooling. With this simplification, there is not that much change but still.
• While we say that TD should be expanded for passing to the binding, I would like to really make sure that a human developer does not need to expand it when used with Scripting API or similar approaches. I think this approach is human understandable enough. What are your thoughts @relu91 @danielpeintner ?
• for reviewers: please ignore everything about schema definitions etc. That is just leftover from TD 1.1.

[benfrancis]

I think this is a huge improvement, thank you for all the hard work @egekorkan. I just have one remaining comment.

I think the case where formDefaults has an array of strings as a value is confusing. It took me a long time to figure out that formDefaults actually means defaultForms in this case, i.e. not the default values for the members of Form, but a default set of Form objects to be applied to every InteractionAffordance.

If this is in fact the intention it raises a lot of questions for me, e.g.:

• The remark next to the formDefaults vocabulary term says "the items have an OR relationship key". In what way is this an OR relationship?
• When a Form in an InteractionAffordance overrides a value, does it always have to override that value in all of the default set of Forms?
• What if an InteractionAffordance has two Forms with different values for the same member, which one wins in the expanded TD?
• What if different protocols support different subsets of operations? Presumably that's not possible with formDefaults, so that term is only useful if all protocols support the same set of operations, or only the default values of op for each Form?
• If there is one InteractionAffordance which should not have one of the default set of Forms, presumably there is no way to express that? So this only works when every InteractionAffordance needs the full set of Forms?
• What if there are no overrides needed to the default set of Forms? As I understand it currently the forms member is mandatory and can not be an empty array.
• If you can have an array of strings, why not also an Array of Objects with inline definitions?

I get why in theory having a default set of Forms is useful when you are using multiple protocols/payload formats/IP addresses and want all affordances to use all of the protocols/payload formats/IP addresses without having to add multiple Forms for each InteractionAffordance, but I think it's confusing and in practice only works for very limited use cases where you have a default set of Forms that apply to all InteractionAffordances and all Forms use the same set of operations, or only the default set of operations for that type of affordance.

Instead of allowing formDefaults to be an array, my suggestion would be that formDefaults (or defaultForm) only contains a single set of default terms (i.e. a single default Form), either as an Object providing an inline Form definition, or a string referencing one of the formDefinitions. These defaults are applied to (but can be overridden by) every Form in the Thing description, even if there are multiple Forms for one InteractionAffordance. If you want to define multiple re-usable Forms you would use formDefinitions.

That would mean that if you have multiple protocols/payloadformats/IP addresses you always need to include multiple Forms in each InteractionAffordance, but at least it is more explicit and makes it possible to add different overrides for different Forms. Currently I think the idea that one Form expands out into multiple Forms if formDefaults is an Array is non-obvious.

If the current behaviour is kept, the term should probably be called defaultForms rather than formDefaults, but I think the questions above would need answering.

Please let me know if I've misunderstood.

P.S. As an aside, I'm curious about the use cases for schemaDefaults, but that doesn't relate to this PR.

[egekorkan]

Thanks for the review @benfrancis . The multi ip address is a use case we have for this feature so that is why it is an array in the first place and going in the direction of the proposal after your questions makes that more difficult to implement. The simple case is indeed more common but that does not make it too difficult to use from my point of view. If we go in the direction, I would propose getting rid of the definitions as well and just allow one default form to be defined. Otherwise, here are the answers:

The remark next to the formDefaults vocabulary term says "the items have an OR relationship key". In what way is this an OR relationship?

A Consumer can use one of the forms. E.g., can use coap or http based on the Consumer's capabilities.

When a Form in an InteractionAffordance overrides a value, does it always have to override that value in all of the default set of Forms?

No.

What if an InteractionAffordance has two Forms with different values for the same member, which one wins in the expanded TD?

No winner :) Both are visible in the expanded TD. This goes in the matrix multiplication discussion.

What if different protocols support different subsets of operations? Presumably that's not possible with formDefaults, so that term is only useful if all protocols support the same set of operations, or only the default values of op for each Form?

See https://github.com/w3c/wot-thing-description/blob/ege-cd-simplification/proposals/common-definitions/tooling/tds.js#L569 (or example 10 if it moves around). As you can see, formDefaults is not used. If there 100 http properties and actions and 1 mqtt event, I think it would make sense to use formDefaults and override for mqtt event.

If there is one InteractionAffordance which should not have one of the default set of Forms, presumably there is no way to express that? So this only works when every InteractionAffordance needs the full set of Forms?

Kind of but you can override with form in a Form.

What if there are no overrides needed to the default set of Forms? As I understand it currently the forms member is mandatory and can not be an empty array.

You would still need the href or a way to identify the resource. In case of WebThing WS Subprotocol, it would look rather empty unless href has a meaning.

If you can have an array of strings, why not also an Array of Objects with inline definitions?

Good point. We can discuss it today. It just adds more variation. We can discuss avoiding formDefintions but that makes it difficult to override. I think there is a tradeoff and the decision would depend on which use case being more common.

P.S. As an aside, I'm curious about the use cases for schemaDefaults, but that doesn't relate to this PR.

That is basically going to be the whole data mapping topic. E.g. all affordance values wrapped in a {value:myVal}

[egekorkan]

Feedback from @relu91 in the call: it would be helpful for this case to allow formDefaults in the affordance level. all properties would have formDefaults:[read,write] and all actions would have formDefaults:[invoke]

[benfrancis]

Or just use formDefinitions instead of formDefaults in this use case...

[egekorkan]

I did not see this feedback before. Could you elaborate?

[benfrancis]

Rather than complicate formDefaults usage further, as a rule of thumb where there are multiple default Forms involved I suggest that a Thing Description author should use formDefinitions rather than formDefaults. In this case there would be one definition for reading and writing properties and one definition for invoking actions.

[egekorkan]

OK I see but without form in individual forms?

[benfrancis]

No, with form in individual Forms (though I think there might be a better name for that term).

[egekorkan]

feedback from @relu91 : What about having href in a form definition and in none of the forms in the compact TD. That would be relevant for WebThing WS subprotocol or any protocol where the resource identifier is the same as affordance name. This can be relevant for defining the default for http binding where the Thing has pattern of properties/myPropertyName. Relevant to address #2087

[benfrancis]

What about having href in a form definition and in none of the forms in the compact TD.

Yes, currently the forms member of an InteractionAffordance is mandatory, can not be an empty array and href is a mandatory member of Form. This means that even if the href is the same as the default for every Form you would have to include a href with the same value in every Form. If formDefaults is specified then maybe forms and href should not be mandatory.

That would be relevant for WebThing WS subprotocol or any protocol where the resource identifier is the same as affordance name.

This is usually not the case in the Web Thing Protocol WebSocket sub-protocol, where the same URL is usually used for all affordances of a Thing.

This can be relevant for defining the default for http binding where the Thing has pattern of properties/myPropertyName

The URL structure is (intentionally) not specified in the HTTP Basic Profile, href can not be auto-generated from an affordance name. I don't think the default HTTP binding should prescribe a particular URL structure either.

[egekorkan]

I am not sure how we can do combo security with the current mechanism. I think going towards how open API defines it might make more sense (object for OR, array for AND). Even that is not possible here. As combo security is not the most common security configuration, I would be ok to make it ugly.

This can be done with introducing oneOf and allOf in the security object and allowing schemes to be put in there. Example that can be in direction:

formDefinitions:{
  base: http://example.com
  security: {
    oneOf: [
      { scheme:basic }, { scheme:digest }
    ]
  }

base is usable in a definition but not in the form level. form is vice versa. Makes validation a bit annoying.

base can be allowed in the form level too

[benfrancis]

I think I may be missing something about the way defaults can be overridden.

@egekorkan wrote:

When a Form in an InteractionAffordance overrides a value, does it always have to override that value in all of the default set of Forms?
No.

How would you override a value in only one of the default Forms from an individual Form instance? How does the Form identify which default Form to override?

What if an InteractionAffordance has two Forms with different values for the same member, which one wins in the expanded TD?
No winner :) Both are visible in the expanded TD. This goes in the matrix multiplication discussion.

Same question as above, how does an individual Form specify which default Form from formDefaults to override?

If there 100 http properties and actions and 1 mqtt event, I think it would make sense to use formDefaults and override for mqtt event.

How would you prevent the EventAffordance being populated with the HTTP-based Form upon expansion?

If there is one InteractionAffordance which should not have one of the default set of Forms, presumably there is no way to express that? So this only works when every InteractionAffordance needs the full set of Forms?
Kind of but you can override with form in a Form.

Same question as above. How would you specify in an InteractionAffordance that one of the default Forms should not be included?

You would still need the href or a way to identify the resource. In case of WebThing WS Subprotocol, it would look rather empty unless href has a meaning.

Right, it's a bit odd in that use case.

Based on my current understanding, in all of the above use cases I would suggest it would be clearer to use formDefinitions rather than formDefaults so that:

1. You can be explicit in an individual Form instance about which base Form you are extending/overriding
2. Default forms are not automatically populated in every InteractionAffordance even if they're not needed
3. A Form definition can be referenced without any overrides

I think this provides weight to the idea that formDefaults should be a single Form, and any use cases where multiple re-useable Forms are needed should use formDefinitions.

Am I missing something?

[egekorkan]

TD Call today:

• @benfrancis it would be quicker for us to explain the mechanism if you join the call. Would next week Wednesday 16:00 CET slot work for you? We can go over your concerns and dedicate the call to this feature. Some answers until then:
  • https://github.com/w3c/wot-thing-description/blob/ege-cd-simplification/proposals/common-definitions/README.md#form here the form is used within a form to override the defaults by pointing to a definition. Presence of a form term in a Form stops the defaults expansion and picks that specified Form definition. Alternatively, you can override a single term. See https://github.com/w3c/wot-thing-description/blob/ege-cd-simplification/proposals/common-definitions/tooling/tds.js#L228 for that.
• We need to note that having no formDefaults but definitions at the Thing-level but then a form not having form key results in a faulty td. This should be added to invalid test cases
• We should provide the expanded versions. Thus, the compact TD examples will have the expanded versions under them. This will change the test mechanism too. I will remove expanded TDs examples. Invalid compact TDs will not have an expanded example.

[benfrancis]

@egekorkan I'm sorry I'm not able to join calls at the moment, but thank you for your response.

Some answers until then:
https://github.com/w3c/wot-thing-description/blob/ege-cd-simplification/proposals/common-definitions/README.md#form here the form is used within a form to override the defaults by pointing to a definition. Presence of a form term in a Form stops the defaults expansion and picks that specified Form definition.

Are you saying that if any of the Forms in an InteractionAffordance contains a form member and formDefaults is an array, then none of the default Forms from formDefaults will be added to that InteractionAffordance when the TD is expanded? If so that was not obvious to me from the current text, and personally I do not find it very intuitive.

[egekorkan]

Are you saying that if any of the Forms in an InteractionAffordance contains a form member and formDefaults is an array, then none of the default Forms from formDefaults will be added to that InteractionAffordance when the TD is expanded? If so that was not obvious to me from the current text, and personally I do not find it very intuitive.

Yes that is pretty much it. Is the non-intuitive part the general algorithm or looking at a TD like this feels weird? To me it looks kind of straightforward. It is the same mechanism as security and securityDefinitions we have at the moment.

[benfrancis]

@egekorkan wrote:

Yes that is pretty much it. Is the non-intuitive part the general algorithm or looking at a TD like this feels weird?

Both - the implicit nature of single Forms that expand out into multiple Forms, and the algorithm for expanding a TD which behaves very differently depending on whether formDefaults is an object or an array, or if the form member is used inside just one of multiple Forms in an InteractionAffordance.

To me it looks kind of straightforward. It is the same mechanism as security and securityDefinitions we have at the moment.

I think the formDefinitions part which behaves like schemaDefinitions (and previously securityDefinitions) is fine. You declare a definition in one place, and explicitly reference it from another. I also think that having formDefaults as a single Form object which defines the defaults for all Form objects is fine as an optimisation.

The parts that I find weird (again, if I've understood correctly) are:

• formDefaults can either be a Form object or an array of strings, but if it's an array it goes from being a collection of defaults for all Forms to being a default set of Forms for each InteractionAffordance (i.e. from "form defaults" to "default forms")
• You can have a TD which appears to have one Form per InteractionAffordance, but actually implicitly has ten Forms per InteractionAffordance which all share one term in common once it is expanded
• You could have a collection of defaultForms which are identical for every InteractionAffordance (e.g. the WebSocket example) but you still have to override something (potentially with the same value), because forms can't be empty...
• If you have a set of ten defaultForms but want to override one term in one default Form in one InteractionAffordance, all of the default Forms get thrown away for that InteractionAffordance and you have to explicitly declare all ten Forms again
• If you have a set of ten defaultForms but you want one InteractionAffordance out of twenty (e.g. your Event example) to omit one of those ten Forms then you have to throw away all of the ten default Forms for that InteractionAffordance and explicitly declare them again. And the only way to do that is what.... by creating a form definition for one of the other nine forms and referencing it inside the InteractionAffordance in order to block the expansion?

IMO all of the above examples would be simpler or more explicit if you just used formDefinitions instead of formDefaults and/or formDefaults was a single Form.

[egekorkan]

@benfrancis I will try to make a proposal in another PR but I have a better feeling about the complaint. Right now, we have the following 3 use cases and 1 is not working well in my opinion neither:

1. Single default form everywhere -> formDefaults as object. This is the most simple case and it is working fine.
2. Multiple default forms everywhere -> formDefinitions and formDefaults as array referencing formDefinitions. In this case, the keys within formDefinitions is useless. It is used for one line and then forgotten. If we make formDefaults an array and not require formDefinitions it would be the same. It would slightly simplify implementations.
3. Most forms same but some variations -> In this case, formDefinitions keys are used but in a bit of a weird way. This is where most of your complaints are and we can think of a better solution.

[egekorkan]

With the latest push, I think this PR is ready to merge. We have two feedbacks from @relu91 and some from @benfrancis that I will move out of this PR as they are not against this but asking for more simplification or usability improvements.

To be discussed in today's call.

[benfrancis]

I'm fine with this PR being merged because I think it's a huge improvement, and it's still just a proposal.

I am happy to file a follow-up issue about further simplifying formDefaults, which is still too complicated in my opinion.

Edit: I have filed #2175.

[egekorkan]

TD Call today:

• The TF is fine to merge this PR.
• The discussion should contain in the issues below:
  • Not using defaults for multiple forms: [Common Definitions] Only allow one default Form in formDefaults #2175 (Needs more concrete use cases)
  • Initial connection oriented protocols improvement: href in formDefinitions  #2176
  • Affordance level defaults: Affordance Level Form Defaults in Common Definitions #2177