Engine/client APIs: Model-only nodes

[eyelidlessness]

As preparation for supporting notes, #174 broke up StringNode into:

• ModelValueNode: any model-only leaf node
• NoteNode: any leaf node associated with an <input> and with a constant-truthy readonly expression
• StringNode: the subset of former StringNodes which are neither of the above

Introducing ModelValueNode was mostly an opportunistic change, refining existing aspects of the engine/client interface to simplify downstream integration where clients need to distinguish between model-only and body elements. Some questions were raised in review of #174 around whether/how to expose:

• Certain computed states on ModelValueNode.currentState:

  • readonly
  • relevant
  • required

• ModelValueNode.validationState

• ModelValueNode itself

• SubtreeNode, which had already been meant* as a representation of model-only structures

* There are probably some edge cases where this is not accurate: some nodes identified as a SubtreeNode may have non-model-only descendants.

When we merged #174, we agreed to defer those questions to this issue. Below will cover the questions raised there, working outside-in from the more fundamental questions to more nitty gritty details.

Long term: [how and when] should we expose model-only details of form state to a client?

Among our current clients:

• The web-forms Vue form entry UI (and to a lesser extent, ui-solid with a similar role) generally has no interest in model-only form state, which by definition is not user-facing.

• The scenario integration test suite does exercise some model-only aspects of form state, but there are some reasons to doubt the "integration test"-ness of those tests.

Imagining some hypothetical future clients, those that seem most likely to benefit from access to model-only state/nodes fall roughly into use case categories like:

• Form design/form builder/etc
• Debug/inspection tooling (whether dev-focused, or overlapping with form design use cases)

Does one representation of form state serve all use cases well?

We might want to think about providing different "views" (in a colloquial sense similar to use of the term in databases, or other overlays of lower-level data like DataView) of form state, serving these use cases with fairly sharp differences in requirements. To start, we could imagine two such "views".

1. One tailored to support the most common user-facing use cases (form filling, submission editing, presentation of submission data in general).

2. A more detailed model, tailored to use cases with a greater focus on aspects of the form itself.

We can also imagine that other concerns fall on either side of the line between these views, such as whether to include details like a node's definition (as we otherwise move to reduce the value of those details in our current focus on 1).

Should we reconsider some integration test boundaries?

Also discussed in comments on #174, there are currently some scenario tests that would fail without access to model-only form state, but such tests' role as "integration tests" seems somewhat dubious, at least insofar as we're mostly focused on testing integration with the most common user-facing use cases.

We may want to consider adapting those tests to either:

• become more explicitly unit tests, insofar as there's value in testing model-only intermediate state tending to be used like implementation details of a form
• restricting integration-level testing of model-only state to that produced in submission data, where that model-only state is represented as values in an XML document

Nearer term: should clients have access to model-only computations?

It seems likely we'll want to live with a single "view" of form state for the time being, as it's working relatively well and not super pressing. Taking that as a near term assumption, and coming back to the first of the two original review comments on #174, it's an open question whether clients should be concerned with the state of core computations on model-only nodes. From a user-facing perspective in current clients (web-forms, ui-solid):

• readonly: model-only nodes should always be treated as read-only, because (as of Engine support for notes #174) those nodes don't expose any write APIs; any explicitly readonly state would only affect whether the engine produces an error when on an unexpected write call (which would almost certainly indicate a bug).

• relevant: model-only nodes should always be treated similarly to non-relevant nodes of any other type, because they are not expected to be presented to the user; any relevance considerations are engine-internal, where non-relevant model-only nodes (and their descendants) are treated as blank in computations (and will eventually be omitted in submission data).

• required: apart from the question of validation (see below), model-only nodes should always be treated as non-required, with all the same reasoning for readonly

Should model-only nodes participate in validation?

As mentioned in #174, required conditions on model-only nodes are not enforced by Collect (or Enketo). I haven't checked, but it seems reasonable to assume the same is true for constraint conditions. We currently deviate on this in Web Forms.

It's not clear as I write this whether that deviation will affect real world forms. Are there real world forms which both:

• actually define constraint/required conditions for model-only nodes, and
• in such a way that model-only data can have validation errors while all user-facing data is valid?

If such real world forms exist, is that indicative of a mistake in form design? If not, what do users (in this case, the authors of those forms) intend, and what behavior do they expect?