CDA JavaScript/TypeScript Client-Side SDK

[krowvin]

CWMS Data API Software Development Kit

Posting here to open discussion on the creation of a browser based software development kit for CDA

Ideally the software would be modeled after the existing DBAPI python api. This would give users of CWMS that already use DBAPI an advantage.

Some icebreakers:

1. Open source - Both public and internal could use this
2. Provides users with a way to focus on their app and not on what fetch() and/or XHR are, what api version we are on, etc
3. Could manage the handling of tokens once auth is completed
4. Has all the endpoints
5. NPM package for ease of use with package managers/projects
6. Would update as changes are made
7. Wraps the DBAPI as much as possible

Name Suggestions

• CDA-SDK
• CDAjs
• ??

Ideally it would be written with typescript, and could be used by both javascript and typescript projects.

Some examples would be:

Getting a catalog of locations

const location = new Location(“SWT”)
console.log(await location.catalog())

Fetching a specific timeseries ID for the last 24 hours

const timeseries = new Timeseries(“SPK”)
const ts = await timeseries.get(“tsid”, begin, end, etc).then(d=>do something)
for (const t of ts.values) {
   console.log(t.name)
}

Other Notes: In theory this could work just fine with NodeJS for server side calls as well.

[adamkorynta]

DBAPI only handles time series and ratings so you might find using that for a template against all endpoints will shoehorn you into certain patterns.

[MikeNeilson]

I like it. Though I think a factory style would work better.

Something like

const api = cwmsdata.for("https://.....");
const ts = api.timeseries("OFFICE");

could even have

const api cwmsdata.for(...).withAuth(<creds>);

That will make it easier to alter exactly which CDA instance is being targeted.

I'll also 2nd what Adam said. I don't think a similar API is a bad idea, but it should be a layer in front of an API better suited to javascript.

[adamkorynta]

Luckily the #1 issue with DBAPI that we keep hearing about is Connection starvation, and that wouldn't be an issue here.

[MikeNeilson]

Famous last words. 😄

[krowvin]

Browser limits async requests to 6 as @DanielTOsborne and I have had fun dealing with.

But as more use it scaling will be fun! ☁️

[MikeNeilson]

huh, I guess that's not too surprising. Though I would've though async requests can go forever... they're async. Or is that specifically if you use "await."

[krowvin]

I think it's actually a limit imposed per browser. Looking at the spec itself HTTP 1.1 may have less, but 2.0 appears to have no limit.

As for fetch and await, in javascript, i have only seen them used to access a Promise which is what fetch returns (if you don't chain). But these let you fire off as many ajax requests as you like in the browser. But only 6 will complete at a time (for Chrome).

All this is moot if you are talking about any other async and/or language. Just talking about browser based apps (which is what the proposed SDK here would be for primarily)

[MikeNeilson]

Also that seems like the kind of thing to put in this repo. since it needs to stay up-to-date anyways. There are several OpenAPI to "client impl" libraries available so it'd mostly be the hec-cwms-data-access esque glue to keep up-to-date.

[krowvin]

If there's a more automated way to transpile for this, i'm all for it! I realize the endpoints are still fluctuating (BETA).

[MikeNeilson]

It's not a transpile. just generation. https://openapi-generator.tech/docs/generators/javascript/

Given our current design it would need to run after the integration tests or, at least it needs to run the "run" task to have an instance.

[adamkorynta]

I see usefulness if keeping an autogenerated library within this repo, but I think you'd want to keep the public interface/factories/etc separate so that you aren't mixing server-side PR's with client side updates and vice versa.

[MikeNeilson]

good point. and that is how the java version currently works.

[adamkorynta]

Using a Fluent API style for this would really make nice examples for users. That's something the DBAPI is lacking.

[MikeNeilson]

Agreed.

[rma-rripken]

The Jython API for CDA that you were referring to is RADARAPI. Its just the time series and rating methods from DBAPI but backed by CDA instead of JDBC. I think your idea is different.

[krowvin]

I'm still not %100 what all DBAPI does. I intended on looking into it and copying method names. I was just under the impression it obfuscated the port/connection/etc away from users.

But realizing from the other thread that maybe this should be its own thing.

I also wanted to make sure someone wasn't already doing this. Because I'd just contribute to their project.

I look forward to doing was RADARAPI has. It's geared towards jython/python iirc?

Whereas I was intending this as something for browser/client apps.

[rma-rripken]

The RADARAPI write side of things is still being developed - I've been held back b/c I've had difficulties setting up the docker integration tests. Mainly its just calling the java cda client code from jython and translating objects

[krowvin]

Just to update folks on where this is. Neilson created a repo here:
https://github.com/HydrologicEngineeringCenter/cwms-data-api-client-javascript/tree/openapi-generator

I was able to generate TypeScript and JavaScript client libraries using a trimmed version of the Swagger Docs API. I need to do more testing with the library and have had issues importing it (something to do with modules / ES6). But it appears to be a solid route and would enable us to integrate it in the build automation for future updates of CDA.

[krowvin]

@rma-rripken @MikeNeilson

When implementing the swagger doc into the openapi generator one issue I ran into was that we have the api path (/cwms-data) added to all the endpoints.

This makes it very difficult to override this in the generated library for the end user.
Looking through the openapi generated source it appears the path is not directly exposed to the configuration only the basePath is.

I propose we remove the /cwms-data basePath from the swagger config and add it into the host.

I manually made this change to my swagger-config.json and ran the generator to confirm it worked.

It is also possible this is a limitation of the generator.
But looking at the generated source it appears to only have a basePath.
There is a path but it looks like it would take extra steps to set the RequestOpts here.

Doing this would alter my #588 to change the servers to this
or
- url: https://cwms-data.usace.army.mil/cwms-data

Example client source after the change:

import { OFFICE, HOURLY_INST } from "../js/constants.js";
import { Configuration, TimeSeriesApi } from "cwmsjs";
import { getDate } from "../js/utils"

const CONF = new Configuration({
  basePath: "https://t7-hostname:tomcatport/district-data",
  headers: { accept: "application/json;version=2" },
});

const TSA = new TimeSeriesApi(CONF);
TSA 
  .getCwmsDataTimeseries({
    name: ts,
    office: OFFICE,
    begin: getDate(-5, `YYYY-MM-DDT${HOURLY_INST.time}:00Z`),
  })
  .then(
    (data) => {
      console.log("API called successfully. Returned data: ", data);
      setTSValues(data?.values || []);
    },
    (error) => {
      console.error(error);
    }
  );

This also could be a knowledge issue on my part as the auto generated documentation also needs some help. (May end up writing it manually, typescript will generate all the docs but typescript-fetch will only make a Readme. )

On the docs, one option is to use mustache/templates. typescript-fetch generator docs