Update API design philosophy

[MikeNeilson]

Okay, so everyone (me too, just worried about the security people) wants a simple "WTF version is this instance", I want to switch to calendar versioning (CalVer) and provide such an endpoint. However, beyond that I think we need to better formalize what our total versioning is since there's definitely been some confusion on that point.

The following will likely read as a manifesto so bear that in mind as at this time I'm not dictating exactly this as the solution but soliciting feedback. So please argue with me, I will attempt to bring references, you should do the same.

Some of this is already stated on the wiki page, but it keeps coming up in various PRs and issues so I think good to bring up a more general case and formalize the docs in a way easier for others to common and propose changes.

There may be differences from https://github.com/USACE/cwms-data-api/wiki/API-Design, That was written a while ago and we have some experience with the API and it's usage now. Anything we decide in this discussion will supersede that document and that page will either be deleted or updated.

---

The general philosophy of CDA

In general

1. The API itself is not versioned. We will use CalVer on the code and provide this on request as in some cases it will be easier to clients to make certain determinations.
2. A URL or URI identifies a resource (Location, timeseries, etc)
3. the authority identifies a CDA instance
4. The Path identifies the type of resource
5. The query parameters identify which of that type is desired
6. a MIME TYPE (Content-Type) identifies the format of a resource, e.g. XML, JSON, JSON with X,Y,Z features RFC2025

Identification of data

1. A URL with names expanded out (e.g. human friendly) is the canonical identification of the resource. e.g /rating/South%20Diversion%Canal%.Stage;Flow.Prod along with any query parameters
2. We will support a more machine friendly ways of accessing data, e.g. something like /rating/8d7197ee-7670-429f-a57e-38abb9c8c4ea. OR some sort of SLUG that internal can used.

Underlying data stores

As CDA is intended to be the primary way into data, it is connected to a database. It is not a goal or requirement of the API to directly map end points to tables, procedures, or views. It is acceptable to do so ; however, the general usage of data expected should be considered and provided in a friendly way that could be rendered as-is in a web page.

Without good reason, we should not expose relational database design constructs through the API. The API exposes Data, not Tables.

Primary target of operations

Usage within websites and system grabbing just data are the primary user base. The design and documentation should focus on those usages, the API itself while providing a simple swagger-ui is NOT intended to be consumed manually. While we should strive to make it easier to do so, it's not the job of the API but systems that use the API on those users behalf.

Versioning of data variable shape data

As stated the data, by content-types, are versioned. In the past there was some severe confusion on this part and it was treated as anything new was "version=2" in the content-type. To allow this design but reduce confusion going forward

1. The initial version of a data set SHALL be the date it was finalized (e.g. PR about to be merged)
2. IF it is the first version of this data the plain content-type "e.g. application/json" will point to this data
3. IF is it not the version version of this data, it will be discussed and announced when it becomes the new default data.
4. Downstream systems SHOULD use the specific version regardless of when implemented, and this behavior should be well documented
5. If a given data set includes definitions of its shape within the type there should be sufficient documentation for downstream developers to properly account for any changes over time. (See our TimeSeries type and discussions within Fixes #634 - Implemented data entry date option for TS data retrieval #927 .

Database interaction design

1. Open a few connections as possible for a given set of operations.
2. If at all possible open a single connection for all operations during a request.
3. If that is not possible, document it for future research.

Searchability (this one is really just a question, not an assertion.)

We need to decide on a single "CATALOG" end point, using the plain endpoint as a catalog e.g. /timeseries, or a catalog for each data type, e.g. /timeseries/catalog

While the current Catalog of time series and locations has query parameters than overlap well, I suspect it will get rather confusing if we add anymore.

@MikeNeilson is in favor of the <datatype>/catalog concept as it makes it explicit.
Alternativaely current /catalog/<datatype> uses a PATH parameter and that could instead just be a path to a separate endpoint. That would group all of the CATALOG endpoints in swagger while allowing them to list and document their own parameters.

Deployment philosophy

CDA should be continuously deployed (at a minimum into test environments) so that all code is affected whenever important changes are made while also not stopping development. As such any endpoint can, and new endpoint must, determine by some method if they can reasonably execute and if not return an appropriate "operation not supported" response to the user.

The result of this check may be cached. A negative results should be cached no more than 1 hour.
Source our eventual version endpoint generate and provide this list? I believe we've all said at one point it's important for systems to be able to determine what's available and give it's a publicly documented and open source API it's not like we're really giving anything away from a security standpoint.

Follow up

After probably both way to much and not enough discussion (RMA folks don't go overboard on the time), I will document the consensus. Probably in a similar, if not identical, style as cwms-python.

Will also separately document anything that still has clear disagreement and requires follow up discussion or more information. Was looking up some of the ADR type docs for OpenDCS and that's not uncommon and it would be could to leave such these in the same place as it would make it easier to move to accepted.

@jbkolze @krowvin @DanielTOsborne @adamkorynta @rma-bryson @rma-psmorris @rma-rripken @zack-rma
@ktarbet @mdstanfill @katfeingold @adamscarberry @jeffsuperglide @rgoss @oskarhurst @jvanaalsburg

Please send anyone who may be interested a link to this to comment on. Above are just the names that have either previously comment on such, may be interested, or whose opinion I value (okay, I value them all. Favor, perhaps?... we all have our biases.)

[jbkolze]

Thanks for writing all this up, Mike -- I'm in agreement with the vast majority of what you have laid out above. I'll try to organize my comments similarly to yours:

• Versioning
  • To clarify, when you say the CalVer can be provided "on request" do you mean an automated request (e.g. an endpoint) or a manual request such as an email? Ideally we would be able to trigger new builds of things like cwmsjs on API updates, so it'd be nice to be able to retrieve the version in some automated manner. Edit: I just saw "our eventual version endpoint" way below so I think I found my answer.
  • Just a minor thing -- if a default endpoint version could potentially change would it make sense to remove the "default" option and always require a version? This shouldn't be an issue if your recommendations (4) are followed, but you know how that goes.
• Data Stores
  • I feel like the current state of the things sometimes puts offering data in a "friendly" manner at odds with 2. A URL or URI identifies a resource (Location, timeseries, etc), e.g. assembling a quick report for latest values of many time series for one location requires a lot of requests from several sources (locations, catalog, time series, etc.). The current paradigm seems to be shifting toward handling this in some cases by shoving things in time series groups, which feels a little messy. This is possibly more an indictment of the database itself rather than CDA.
  • Is the idea for every "resource" to map directly to a database object/type?
  • I hear a whole lot of talk about doing things like running cron jobs each night to pull POR data from CDA, generating graphs in python, and uploading to S3 (or worse, GitHub) for web viewing. This feels pretty antithetical to the preferred path forward. I wonder if things like this could be made more "friendly" by striving toward "on-the-fly" simplification of huge datasets, e.g. requests for 50 years of 15-minute elevation data would be resampled to 1-day by default unless explicitly overridden.
• Searchability
  • <datatype>/catalog feels better to me. The current implementation (datatype as query parameter) turns into a huge mess in typed languages, e.g. TypeScript. And honestly it's probably a mess in JavaScript as well -- you just don't find out until it blows up at runtime.

[MikeNeilson]

Excellent feed, I think we're actually in near complete agreement here but the language I needs improvement.

Thanks for writing all this up, Mike -- I'm in agreement with the vast majority of what you have laid out above. I'll try to organize my comments similarly to yours:

* Versioning
  
  * To clarify, when you say the CalVer can be provided "on request" do you mean an automated request (e.g. an endpoint) or a manual request such as an email?  Ideally we would be able to trigger new builds of things like cwmsjs on API updates, so it'd be nice to be able to retrieve the version in some automated manner.  Edit: I just saw "our eventual version endpoint" way below so I think I found my answer.

Just to be super clear, definitely and endpoint. People have to ask now, it's super confusing.

  * Just a minor thing -- if a default endpoint version could potentially change would it make sense to remove the "default" option and always require a version?  This shouldn't be an issue if your recommendations (4) are followed, but you know how that goes.

We have, though it's pretty much why we're here. That said, we could just say "we're not going to support direct browser usage and and if a plain content-type is present do like a redirect or something, but the "default" is mostly to support that copy paste... which now that I type that is pretty much the cause of people not putting proper header in (which browsers don't allow properly.)

* Data Stores
  
  * I feel like the current state of the things sometimes puts offering data in a "friendly" manner at odds with `2. A URL or URI identifies a resource (Location, timeseries, etc)`, e.g. assembling a quick report for latest values of many time series for one location requires a lot of requests from several sources (locations, catalog, time series, etc.).  The current paradigm seems to be shifting toward handling this in some cases by shoving things in time series groups, which feels a little messy.  This is possibly more an indictment of the database itself rather than CDA.

I don't think data so much as the disconnect between how the database or CDA provides things and how they're used. Some of it I think we can fix over time.

  * Is the idea for every "resource" to map directly to a database object/type?

The opposite of that. For some things it certainly makes sense, locations being locations, but we're already doing some "virtual" things with gates/outlets as it's cleaner. Somewhat in relation to you're above comment, if a set of data is constantly being used together, can be well defined (the DTOs) and there's a good way to allow the user to make the request (inputs) we may as well make an endpoint to allow it. Example, the Location Levels if using a time series backing are a composite of the Level data and a TimeSeries DTO object.

  * I hear a whole lot of talk about doing things like running cron jobs each night to pull POR data from CDA, generating graphs in python, and uploading to S3 (or worse, GitHub) for web viewing.  This feels pretty antithetical to the preferred path forward.  I wonder if things like this could be made more "friendly" by striving toward "on-the-fly" simplification of huge datasets, e.g. requests for 50 years of 15-minute elevation data would be resampled to 1-day by default unless explicitly overridden.

This is a difficult one. I don't agree with such on the fly resample, plus actually sending the data isn't an issue, transferring data isn't an idle connection so it can keep running. But to even do that subsample requires either pulling all of the data or writing the sql to do it (granted trivial in the subsample case) but the system still has to wait for the data to be gathered to start sending and even the best system will likely not meet our need; especially when date-versioned time series are considered.

* Searchability
  
  * `<datatype>/catalog` feels better to me.  The current implementation (datatype as query parameter) turns into a huge mess in typed languages, e.g. TypeScript.  And honestly it's probably a mess in JavaScript as well -- you just don't find out until it blows up at runtime.

Wasn't even thinking out that, but also a fair point.

I don't think any of those are really a counter point, but hopefully just adds some clarity to my initial thoughts. I also look forward to seeing what everyone else comes up with.

[krowvin]

e.g. requests for 50 years of 15-minute elevation data would be resampled to 1-day by default unless

I know we have gone round and round about this. And I know cache is implemented now to some degree.

• Implement a Caching solution or at least a framework for one #532

I still think an edge cache like (Redis/Memcached/etc) could solve this if not just for POR data. I.e. explicitly precaching POR (not just per request)

If we know the definitive data sources that will be pulled frequently, then having them cached on some interval in a memory store would certainly benefit things like pulling 50 years. Then take a step further to allow CDA to query from this first and pull bits from this. (for example if 5 weeks is requested of 50 years of data, it could still yank from the cache (see Redis TimeSeries)

I know A2W new API is doing this with levels now.

But for some fresh perspective, what are your thoughts on edge caching @jbkolze?

[MikeNeilson]

I'm in favor of caching over a automatic resampling solution. How one resamples data really depends on the data. E.g. is it an instanteous value, already averaged, a total, what's the window of non-instananeous? We'd end up recreating a rather large portion of DSS-math within a simple retrieve function and likely be wrong for the given usage (maybe the person pulling POR is wanting to experiment with such.)

Though I suspect any discussion more detail than what we've done should be in the linked discussion (there's no rule we only need to cache data in one way.)

[jbkolze]

I think caching is a good idea as long as we can work out a consistent way to identify the frequently-accessed portions of the data (maybe primarily associated with the official/virtual time series records once they're implemented?).

That being said, I'm not sure that will fix the potential problem I have in mind. I foresee a situation in which someone wants to create an annual plot of a 5-minute time series and, given that JS plots aren't the most lightweight of components, the resulting ~100,000-point plot crashes or at least severely bogs down the client-side browser. Ideally you'd display less points, but in the current paradigm that means setting up CCP computations to resample the data specifically for plotting. And to support various time windows you'd need to create multiple time series with various intervals.

Agree with Mike that making resampling assumptions could get messy, and I keep forgetting that our speed limitations are more so on the database side than the transfer side. I don't have a great answer right now, but I do see it as a potential concern just due to the way that we and our users interact with the data. This might be one where the solutions become more apparent once more districts dive in and specific use cases are revealed.

[MikeNeilson]

or at least severely bogs down the client-side browser

I'm going to go out on a limb and suggest that the API can't, and should not even try, to fix excessively silly downstream behavior.

Instead we should document what we support, why, and leave it up to those usages to sort out the details. Otherwise we're just going to chase everything, and get nothing done.

[krowvin]

Likely include capability list or matrix.

This is great, and could allow client-side software to switch between versions of the server. I.e. enable/disable client-side features based on the target server's resultant list.

Ideally have a verbose and minified version of these features. (URI Param?)
I do not know if a hash is useful or not here. But human readable vs non.

Also +1 to everything Brandon said

[adamkorynta]

CDA should be continuously deployed (at a minimum into test environments) so that all code is affected whenever important changes are made while also not stopping development. As such any endpoint can, and new endpoint must, determine by some method if they can reasonably execute and if not return an appropriate "operation not supported" response to the user.

This alludes to a lack of database support (i.e. forecast API updates). I believe the comment is intentionally vague to not limit to just the database schema version (such as message queue support), but for general dissemination I think including a sentence to call out the database is worthwhile.

I'm also interested if the eventual goal, given CDA will be the only access to the database, is that database schema would also be continuously deployed with along with CDA.

[MikeNeilson]

For the continuous deployment of the database the answer is that it's the long term plan. But our current build system doesn't make it very practical hence the migration to flyway to handle the schema migrations.

[adamkorynta]

We need to decide on a single "CATALOG" end point, using the plain endpoint as a catalog e.g. /timeseries, or a catalog for each data type, e.g. /timeseries/catalog

I don't really understand the catalog endpoint and would personally vote for its removal. It seems like all other datasets created after the initial catalog was created started to follow the Javalin CrudHandler format (while I don't think our API design should be constrained to a framework design) where each endpoint has getOne and a getAll methods. The timeseries catalog can be replaced by the /timeseries/identifier-descriptor/ getAll method endpoint. The location catalog can be replaced by the locations getAll method endpoint. All of the other "cataloging" mechanisms already follow this paradigm and I don't see the upside of moving them to something like /blob/catalog or /catalog/blob.