Offline entity usage metrics for 2024.2

[ktuite]

Closes getodk/central#653

This covers these metrics:

• 1. Number of offline branches involving more than one update
• 2. How many branches were interrupted in the middle of being processed?
• 3. How many submissions were put in the backlog just long enough for updates to be processed in the correct order? In other words, how many submission.reprocess events have there been?
• 6. How long does it take for an entity change to be processed? What is the max amount of time between when the submission was created and the entity version was created?
  • Matt: "What I'm personally interested in is some sense of how often there are long delays in entity processing due to a large number of unprocessed events in the audit log. How often are the workers working through the audit log overworked?"

It doesn't include

• 4. How many offline updates have been force-processed?
  • This requires changing entity audit logging a bit to be able to track this (Log when a create/update entity submission is force applied central#720)

TODO:

• tackle this one:
  5. How long does it take for an entire branch to be processed? Specifically: what is the max amount of time between when the first submission for a branch is received and when the last submission is processed?
• Hook up the rest of the analytics (send output to odk form, update analytics form, etc.)

What has been done to verify that this works as intended?

Why is this the best possible solution? Were any other approaches considered?

How does this change affect users? Describe intentional changes to behavior and behavior that could have accidentally been affected by code changes. In other words, what are the regression risks?

Does this change require updates to the API documentation? If so, please update docs/api.yaml as part of this PR.

Before submitting this PR, please make sure you have:

• run make test and confirmed all checks still pass OR confirm CircleCI build passes
• verified that any code from external sources are properly credited in comments or that everything is internally sourced

[matthew-white]

Looking good so far, just leaving some initial thoughts!

[matthew-white]

Could this exclude offline updates that are force-processed from the backlog? I think those would skew the metrics.

[matthew-white]

Maybe this is another place where getodk/central#720 would be helpful.

[matthew-white]

Based on the discussion below, I think this isn't an issue after all. Even if an offline update ends up in the backlog, that's different from how quickly the submission.create event is processed.

[matthew-white]

For the same reason, it'd be great if we could exclude submission.create events that were processed when the approvalRequired flag was toggled, as we expect a delay in that case. I'm not really sure how to identify that though. 🤔 If it helped, I think it'd be OK to exclude entity creations (entity defs whose version number is 1), since those are the only entity defs that the approvalRequired flag will affect.

[ktuite]

This query is kinda indirect in a way that I think is convenient for the metric we are getting at.

It's looking at the timestamps on the submission.create audit event which only gets updated by the jobs worker, the one that runs the first time the event comes in. The other things that process submission events to make entities (like the batch processing when toggling the approvalRequired flag and batch-processing the backlog) use the event to look up information, but they don't update anything about the underlying event.

So I think it's safe to to look at this time difference like this. It's not every submission create event, it's only the ones that ultimately turn into entity versions. It doesn't capture how long it takes overall for a submission to become an entity (potentially factoring in these other sources of delay), but it does capture how long a submission would be sitting around waiting for any initial entity processing (even if the result of that was it getting skipped to be reexamined later).

[ktuite]

Maybe what you were expecting was a more explicit (entity def creation date - source submission def creation date). I could add a query for that, too, that could include these other ways in which entity processing can get delayed. Then we could see if there's a noticeable difference... which would mean these delay-related mechanism are activated.

[matthew-white]

It's looking at the timestamps on the submission.create audit event which only gets updated by the jobs worker, the one that runs the first time the event comes in.

Ah that's interesting, I hadn't put that together. This sounds like a good way to approach the metric then. 👍 I don't feel like we need anything more explicit: it makes sense to me to start with this.

[matthew-white]

I think metric 5 is also one that would benefit from getodk/central#720.