Event-driven Data Feeds (Pub/Sub)

[jbkolze]

I'm curious about long-term plans for event-driven dissemination of data which may only be updated once a day or on an ad-hoc basis.

Currently, CDA has good coverage for "passive" dissemination of data. i.e. cases where users actively reach out with data requests. However, many WM products (such as the aforementioned daily forecasts) may be published only intermittently and ideally would notify stakeholders when new data is available. In many districts, LDM or emails are currently used for this style of data dissemination.

While intermittent data could potentially be monitored by users by continuously polling CDA, I assume that wouldn't be ideal from a traffic standpoint. Are there any long-term plans to introduce some sort of pub/sub model (e.g. webhooks) to CDA? Or, will LDM (or another solution) continue to be preferred for this particular use case?

[MikeNeilson]

The plan actually exposes queues/topics of data. An initial test was done here: #698

Unfortunately, the current design (especially auth) doesn't lend itself well to exposing those systems... and there's whether or not to link to the in database queues or have CDA itself focus on only handling the "cloud queues" - for lack of a better term.

But that's an interesting idea. I hadn't thought of Webhooks, NOAA is wanting to do that for the DADDS replacement. That would either require allowing those wanting the data to signup to manage the hooks, or the districts doing it on their behalf.

So my current thoughts:

short-term

We need something running in our cloud area that listens to the queues and then sends. either Email, LDM, or other.
(This technically doesn't go away, they just change to not directly connect to the database)

mid-term

RSS or ATOM feed?

However, this really only seems most useful for things like a daily roundup of Location metadata or rating changes... though forecasts aren't every minute so this would work there too.

long-term

We have the queues/topics (sorry, topic -> pub/sub, I'm using the ActiveMQ and other systems terms) exposed to all. (With appropriate rate-limiting for unauthenticated connections.)

(We also want this for say OpenDCS and Compproc as well as other tools so it's only long term due to complexity)

[MikeNeilson]

I did a little experimenting, unfortunately it doesn't work quite as nicely as I'd like with the current version of Javalin (once we can upgrade it does work though) but I think I've realized I've been overthinking some use cases.

I was able test using Server Sent Events as a "broadcast" (e.g. not having to retain a cache of event per user). It wouldn't be too difficult to bridge the current, and future, queues into this system and send those events to clients. All the browsers, and cURL, support this.

One reason to do this over something more complex, I really couldn't think of a reason anything other than CDA would need to actually put an event in the queue. Even say OpenDCS wouldn't put an event in that queue/topic it would just save data and CDA itself would handle that.

There's still some security issues to try and sort out (non-auth users shouldn't get everything even though there's no data) but it seems like that would work for a lot of use-cases.

I still also like the idea of an RSS feed that's updated daily, not everyone wants/needs things in real-time.