Form Challenge

[timneedham]

The Forms Challenge

Category

• Data
• Document
• Technical
• Other Suggestions

Challenge Owner

West Midlands Fire Service

We're a small in-house team within West Midlands Fire Service that has over 12 years experience in developing dynamically-rendered input forms (i.e. a technique that separates the definition of what a form should collect from the code that controls how the user interface is delivered).

This separation-of-concerns has worked really well for us over the years and continues to bring unexpected benefits. We've now accumulated a library of around 100 form definitions which cover all aspects of Fire Service activity.

As we begin work on our next-generation platform we'd like to align to a standard that helps widen the uptake of this approach.

Short Description

This is a challenge to produce an open standard to express user-facing input forms.

The standard should cover several facets:

Facet	Description
Layout	The order and configuration of UI widgets.
Binding	How UI widgets relate to an underlying data model.
Appearance	Prompts, labels, grid arrangement, iconography, styling-overrides etc.
Structure	Arranging UI widgets into sections, groups etc.
Enumeration	For populating drop-down lists and similar.
Context	The specification should be expressive enough to indicate how a form should behave/appear in different contexts (in-the-field, in-the-office etc.)
Validation	Min/max ranges, required/optional attributes, [regular] expressions, function-binding etc.
Dynamic content	Use of REST APIs to get content/values and perform server-side validation, "typeahead" support, UUID generation etc.
Behaviour	Conditional appearance of items/groups/sections, enumerated values, read-only states etc.
Nesting	Allow for repeating groups of UI widgets (with min/max allowed number of entries and similar).
Advanced	Internationalisation, scripting support, tours, offline-fallback configuration...

Along with our prototype DSL here are a couple of established reference-points to start:

• React JSON Schema Form
• Schemaform.io

Note!

• This standard should obviously be technology-agnostic (i.e. it has absolutely no relation to any front-end technology such as React, Angular or any other)
• We'll be suggesting related "Data Model" and "Workflow" challenges soon!

User Need

As this is a more general challenge, it doesn't relate to one particular group of user.

That said, what is important from a user-perspective is that the specification be expressive enough for dynamic form renderers (be them running inside web pages or mobile apps) to deliver rich and efficient user experiences.

Expected Benefits

Assuming this standard could be used throughout government to define user-facing forms...

Benefactor	Benefits of a successful challenge
Users	All forms would be consistent (at least within a single platform/product which delivers a set of form definitions) meaning one interface to master and less systems to learn. Users can expect a greatly improved data-collection experience as any effort to improve a generic form renderer would then directly benefit all forms (and by extension, all users). If a form is being delivered from a generic platform, then it is safe to assume authentication is happening across all forms - reducing system-switching burdens
Operational	Assuming a tool-ecosystem built around this standard: organisations will be much better positioned to set and refine their own data collection requirements, making for a more responsive and agile government. Support for deep validation, tours, keyboard-accelerators and other generic functionality will help drive-up general data quality and efficiency. If form content is delivered via a generic platform then authentication and authorisation management would be centralised. Government interoperability and transparency will be greatly improved.
Social	This specification (combined some other challenges such as data models and workflow) begins to pave-the-way to a much more collaborative and open approach to assembling software - akin to the benefits attributed to low-code platforms.
Environmental	When combined with quality tools, the ease of replacing paper forms with electronic equivalents may drive-down paper consumption?

When quantifying the impact of introducing our initial dynamic-form platform, we internally estimated that it was saving 15,000 person-hours per year (as compared with the overheads incurred with traditional "discreet system" approaches and training). If such estimates still hold true today, then the cumulative impact of supporting a switch to dynamic-form-rendering across government would be significant.

Functional Needs

We consider the proposal should be:

Need	Description
Agnostic	The specification should be independent of any technology or vendor.
Lean	Not full of cruft, use intelligent defaults, JSON-over-XML etc.
Intuitive	Needs to be logical: easily read and understood.
Extensible	Over our 10 years, we've accumulated a palette of some 30 different UI components (covering the obvious text-boxes through to gazetteer-selectors and maps). However, the ability to express unforeseen specialist widgets would be required.
Toolable	To deliver wider benefit, the specification will need to play nicely with IDEs, WYSIWYG editors and similar.
Support inference	Meta information, such as prompts, descriptions, data types etc. can be inferred from an associated data model definition (soon to be the focus of another challenge!) As such, the specification should define explicit/predictable behaviour when inferring values.

[edent]

Thank you for this detailed proposal. I'm going to ask @psd to see how this fits in with the work we're doing on forms.

Regarding the two schemas you've mentioned (Mozilla's and Angular) - are there any compelling reasons to develop your own competing schema? We usually encourage people to work with existing bodies, rather than creating a new standard.

Both have open source licences, which is a good start for us.

[timneedham]

Hi @edent

Thanks for your quick response!

Yup, we used the Angular project in recent prototypes: and it's worked really well... we got the base capabilities up-and-running in no time.

However, there are many areas where we feel both these projects lack the capabilities of our existing DSL (and certainly some of the ideas we'd like to introduce in the next!)

Both projects are also tied to the strengths/weaknesses of a particular front-end technology; fracturing things from the outset?

We were hopeful to find a fuller/richer standard that is technology-independent too?

[edent]

Our first questions on standards like this is....

1. Can you contribute to the existing standard(s)? If your new DSL is useful to you, it will be useful to other people.
2. Of the standards that you've seen, do any have a good track record of accepting changes?
3. Do any of the existing standards support extensions? That is, you use their standard with your own, well-defined, addons.
4. Do you have the capability to run a standard better than the existing bodies?

Those aren't meant to be aggressive - just need to understand where it makes sense to collaborate.

[arkie-boy]

I'm the editor of an ISO standard for the representation of the design of forms as metadata that is relevant to these requirements. Standards from our working group are freely available - although I think you can still pay for a copy if you like! - so it meets a practical definition of 'open'. The standard describes a model for registering forms, sections, questions and their answers together with the form's logic in a platform independent way that may be used - with suitable templating code - to generate web forms for data entry. Personally I have transformed form definitions into W3C XML Schema and then into XForms, but I don't see any reason why the same cannot be achieved for other form languages.

We're about to move from a technical specification to a full blown IS: I'd be delighted to talk about your application and (hopefully) ensure that the standard accounts for your use cases, I would be very grateful if we might use one of your forms as an example if there are no copyright issues.
19763-13-PDTS.pdf

Best place to start is reading appendix B

[edent]

Thanks @arkie-boy - could you give us a link to your working group?

The licence for West Midlands Fire Service appears to be MIT https://github.com/wmfs/tymly/blob/master/LICENSE

[arkie-boy]

I'll have to check how the MIT license works with ISO.

The working group is ISO/JTC1/SC32/WG2. We're best known for ISO11179 which is a standard for metadata registries (enterprise data dictionaries, basically).

https://www.iso.org/committee/45342.html

Steve

[timneedham]

To answer your initial questions @edent ...

Can you contribute to the existing standard(s)?

• Not in any satisfactory way really. The closest examples we've found (SchemaForm.io and React-Schema-Form) aren't standards as such, but proprietary formats that support (really well) their own open source projects. As it stands, we'll need to pick a project to contribute our suggestions to - and therefore binding ourselves to a specific front-end technology.

If your new DSL is useful to you, it will be useful to other people.

• Absolutely, that's why we thought we'd start the conversation here! :-)

Of the standards that you've seen, do any have a good track record of accepting changes?

• As above, they're not "proper" standards really, their general issue-turnaround seems pretty good:
  • https://github.com/json-schema-form/angular-schema-form/issues
  • https://github.com/mozilla-services/react-jsonschema-form/issues (gone a bit quiet recently?)

Do any of the existing standards support extensions? That is, you use their standard with your own, well-defined, addons.

• No (there are mechanical ways to add new framework widgets, but not at the "specification" level).

Do you have the capability to run a standard better than the existing bodies?

• Not really, but it is something we're passionate about! That said, we thought suggesting a challenge might be a pragmatic first step to see what ideas are out there? Already the links from @arkie-boy are something we've not picked-up on and look really interesting.

Those aren't meant to be aggressive - just need to understand where it makes sense to collaborate.

• Not at all, very much appreciate all your help! :-)

[psd]

Hi Tim, that's very interesting and we're definitely 'fellow travellers'. I'm away on holidays for the rest of August, but I'll definitely take a look at the tymly code and see how it works. It would be great to see if we can work on a challenge together in the Autumn as we have a similar need for an open standard.

We (GDS) recently ran a discovery into how teams design, publish and process offline forms across government, some of our findings will be made public soon.

During the course of our research we spoke to a number of different teams who are mechanically generating online forms across government using a number of different technologies, including de facto formats such as Adobe XFA, and a number of DSLs in JSON and YAML, notably the Home Office Forms (HOF) and Digital Marketplace.

We're in the process of working out what ideas we could test in an alpha (should we go ahead), but something I definitely see as being useful is a format for describing a form which we can use to consistently model forms and possibly transpose to and from one of the many other formats used to describe existing forms.

Some of the initial needs we have for such a format include:

• being open, ideally an open standard
• able to compose a form from a set of common questions, reusable across multiple forms
• able to use GOV.UK patterns and elements from the for online versions of a form, such as having a page-per-question, using accessible country, address and other pickers where available
• generating PDF/A or HTML with a robust print stylesheet for offline versions of a form
• supporting mutli-lingual questions and guidance

but there are many other needs which we imagine will emerge during an alpha, including some of those you enumerate.

Paul

[timneedham]

Hi all, thanks for all your input, sounds exciting!

• We'll attach our current XML Schema (that's used to render our current-gen dynamic forms) ASAP.

Just a quick question: as above, we've a few other interrelated challenges waiting-in-the-wings:

Challenge	Description
Data model	A discussion about how best to extend JSON Schema to include extra attributes (such as which properties are good for indexing and similar).
Workflow	We're finding a lot of benfit in using Finite State Machines to run workflow, but there doesn't seem to be a standard way to define machines/states/events in JSON? Full-fat BPMN, BPEL etc. seems overkill for our needs.

• All these bump into each other... I guess it would be best to create them as separate challenges and cross-reference them?

[edent]

@timneedham Separate challenges are fine - but we will ask you to take on some of the discovery work.

[jazz-b]

Hi everyone, this is the workbook schema file mentioned above...

workbooks.zip

[timneedham]

Hi everyone,

A quick (and surprisingly emotional 😳) update...

In the end we delivered our MVP using React Schema Form as the basis of specifying UI content declaratively. It went OK, but it's not a great fit. We've had to extend much more than we expected and it's already starting to creak with pretty simple UI descriptions. That said, our MVP has really helped us discover what it is we really need...

We've recently started turning our requirements (for a language capable of describing UI content for use within a digital service) into a specification along with a supporting SDK. We've called it...

✨ Viewscript ✨

Maybe it's because we've been at this for ten years (and Viewscript was always hiding just under the surface, like an open standards truffle)... or perhaps we've some leftover endorphins from getting our MVP shipped... but we're really quite excited about all of this. It feels like a lot of good things can come from something like a Viewscript specification.

Viewscript links

Note that things are less than a month old at the moment and so very much incomplete!

• Viewscript documentation - We're coding Viewscript in-the-open, and all that.
• Viewscript motivation - How we ended-up developing Viewscript, when we really tried not to.
• Viewscript playpen - Use the editor to write some Viewscript and hit "Go" to take it for a spin (it's not the best UI ever, but the various preset "Examples" - especially the "Complex Example" hopefully provides a decent impression of what it is we're trying to achieve?)

Apologies if this is bad form...

Any feedback/discussion/exploration/advancement of the team's work would be really, really appreciated! 😊

Kind regards

Tim

[timneedham]

We're seriously considering using AdaptiveCards as the basis for our declarative front-end UI.

The Guiding Principals, especially 7 are quite heartening and suggest we can extend towards something akin to Viewscript:

The format needs to be extensible

Hosts should have the freedom of adding support for custom elements or custom actions that go beyond what the format is capable of
This is particularly important for actions, as various hosts don't necessarily support the same set of actions
These additions are at the discretion of the host
They are not a de facto addition to the Adaptive Card specification
As such, they make a payload that uses them incompatible with the mainstream Adaptive Card format
They can however be presented to the Working Group and proposed as new features for a future version of the format

Does anyone have any thoughts/experience re. AdaptiveCards at all?

[timwis]

👋 hi folks, this is a topic I've researched a bit before for a few projects as well. I found that the existing implementations out there weren't all that standardised, and often worked around JSON Schema, which makes sense for validation, but doesn't support things as essential as ordering the fields, conditionally displaying them, etc. which resulted in essentially duplicate/redundant configs (one for the "schema" and one for the "display"). Maybe there's no way around that, but I'd love to dive deeper to find out if there is. (Part of me does wonder whether at the end of the rabbit hole it's just HTML :O)

Anyway, just wanted to connect this thread to @dmethvin-gov from the US Digital Service, as they were working on usds/us-forms-system which also used the mozilla approach.

[timneedham]

Hi @timwis , thanks for this!

We'll take a look at the US Forms System.

Not sure if it's of interest, in the end we had a go at the stuff you've mentioned... conditional visibility, validation etc. Based on a heavily-extended AdaptiveCards spec.

We'd love to know your thoughts about Cardscript?

Playpen:
https://wmfs.github.io/cardscript/

WIP docs
https://wmfs.github.io/tymly-website/reference/#cardscript

(take a look at the complex and Kitchen sink examples?)

[galund]

It's worth mentioning that react-based systems aren't readily compatible with the service standard. The service manual has just been updated with regards progressive enhancement: https://www.gov.uk/service-manual/technology/using-progressive-enhancement

[timneedham]

Good point @galund! 😄

In our particular case, the Vue framework we're currently using with Cardscript (Quasar) supports Server Side Rendering, which may help allay some concerns?

This standard should obviously be technology-agnostic (i.e. it has absolutely no relation to any front-end technology such as React, Angular or any other)

Of course, any declarative [JSON] standard could also be used as the basis to generate flat HTML too... the separation of a declarative standard and the technology which may subsequently visualise it is a core principle of projects such as AdaptiveCards and Cardscript.

[timwis]

@timneedham Cardscript looks awesome! Really similar to what I had been toying with. I think I gave up whilst trying to safely evaluate user input for conditional visibility expressions which I see you've managed to wade through :P

[dmethvin-gov]

Thanks for the ping @timwis!

I can't get the CardScript site to load, I'm getting this error and a blank screen:

vendor.373f220d.js:6 Error: Loading chunk 149bcf13 failed.
(error: https://wmfs.github.io/cardscript/js/149bcf13.dc65cbf2.js)
    at HTMLScriptElement.l (runtime.6446a868.js:1)

After quite a bit of hands-on experience with forms, I think that the approach taken by React-JSONSchema-Form (RJSF) is not really a good one. Although JSON Schema is great for defining the resulting data of a form and the constraints on its data, it doesn't have the expressive power to define all the semantics that are needed in the front-end. Concepts like this:

• Fields that should only be displayed when certain conditions are met
• Presentation to indent or highlight fields to clarify they're dependent or grouped
• Validation and when it occurs (As you type? When focus is lost? Just before submit?)

RJSF tried to do this with additional rules "alongside" the JSON Schema but this tended to make the definition very redundant because many fields were mentioned in two (sometimes three or four!) places. Also, it is much too tied to React when it doesn't need to be, as already mentioned. So much of the definition of the form has nothing to do with whether it's in React, Angular, jQuery, or plain JavaScript.

There is prior art here that seems pretty close in concept, see https://github.com/formio/formio and its video at https://www.youtube.com/watch?v=d2gTYkPFhPI . It's not perfect but it does have a visual form builder and creates a "schema" for the form in JSON. From there you can run a builder that generates the resulting JSON Schema for the backend and components for a framework like React, Vue, or Angular.

Beyond that I'd love to see any form system be able to pull in different widget sets via some "adapter" mechanism. It's waayyy too heavy a lift to define all the different types of widgets that people need, and there are several good sets out there already for the popular frameworks. To me that would be the primary reason to use a framework and not just have the form system do its own state and form management in plain JS. I don't want to build yet another set of date inputs, date picker, date time picker, etc. This is especially painful in React where there isn't even agreement on how to manage CSS in an app.

I feel like our ideal here should be to avoid the need for experienced programmers for something as rote and repetitive as form creation. In the US federal government there are more than 25,000 paper forms. We can't expect to automate those in reasonable time if we require a programmer to write each one.

[timwis]

Well summarised @dmethvin-gov - I've run into the same issues! The cardscript playpen loaded fine for me - give it another shot? I think you'll find it more closely aligns with what you've described.

[DidacFB-CDDO]

As per housekeeping practices we are closing this suggestion down since there has been no comments since 2019.