Create Endpoints for Water Management Reports

[krowvin]

Water Management Districts create PDF, CSV, Text, HTML, etc reports for upward reporting, archival, and public display.

These reports are usually monthly, hourly, yearly or other varying data intervals ( See List of Reports below)
We need to be able to store these in a centralized place for viewing on websites. I know of at least one district that has opted to use the BLOB/CLOB endpoints to do this.

While you could create a flag to decide when the BLOB has a backend for S3, in this situation it might make sense to have a separate endpoint since this is a very targeted topic.

Create new Reports endpoints

I propose, after a call with the reporting team, that we create a /cwms-data/reports endpoint for viewing, updating, deleting, and creating reports.

S3 Backend

These endpoints would use an S3 key to talk to the same bucket that @willbreitkreutz manual report upload site is using.

• Storage is expensive for Oracle/DB
• Using S3 we would be able to access the same data/reports others are for various apps
• Improved security, having CDA manage the S3 bucket key would create the necessary security buffer between S3 storage and end developers/users
• Allows easily written web apps that can take advantage of CDA for reports cwms-python/cwms-js

508 Compliance

I suggest we create a flag, or through the filename, show which reports have been 508 scanned.
The filename, is a cheap way to do this with something like "my-report.scanned.pdf"
But could be easily broken, and gets messy.

I think a better way to do this would be to have CDA, or some other process, manipulate the metadata of the document and add a flag in the document (PDF). Then the various applications (CDA/manual report site) when reading S3 can read for this flag to know if something has received a scan.

This metadata approach does not work for plaintext files of course..

Suggested Endpoints

GET: /reports/

• fileName:
• fileType
• author
• agency
• description
  Where any of these could be wildcarded

POST: /reports/

parms:

• *fileName: myreport
• *fileType: pdf
• *agency: SWT
• author:
• description:
• content: base64 encoded?

DELETE /reports/

• *fileName: wildcard
• fileTypes: pdf,csv,etc - comma delimited
• *begin: ISO
• *end: ISO

Where * means required fields

Features

More endpoints could be created to return things like reports requiring compliance checks
i.e.

GET /reports/508

Would return a list of reports that do not have the 508 flag set or compliant: false is set in the docs. This may be difficult without database help due to the time it would take to read metadata of MANY reports.

The GET endpoint should not require Auth.

If a report requires it be sensitive and not exposed publicly then perhaps the manual report website could store to a different S3 bucket/location for those documents?

List of Reports

EM 1110.2.3600

• Conditions
• General
• Project Operators Reports
• Water Control Morning Reports
• Special Advisories
• Discharge Data
• Flood Damages
• Reports for the Media

[MikeNeilson]

Point of clarity. Are you intending this as a "convenience" endpoint to clear semantics for retrieval and NOT general the reports?

It looks that way but I'd like to confirm.

[krowvin]

What do you mean by NOT general? Assuming you mean policing/ generaling in direction

Yes to convenience. Lets us easily wrap the S3 API. I'd say it's 80% convenience / 20% policing/generaling

This is intended to be the primary means to placing reports into cloud/S3.

The manual reporting tool would be the other way, unless of course someone else had an S3 key to also talk to this bucket?

We talked in the past about having the BLOB/other endpoints have S3 backend, but for this situation not sure if that makes sense.

[MikeNeilson]

.... ugh, "generate". sheesh, I can type.

[krowvin]

Ah! Then yes purely convenience.

Users would generate the report and use CDA to store it and then later read it.

[MikeNeilson]

This doesn't seem unreasonable to me. For S3, one of our plans is to move the blob/clob backend to S3 but it will likely wait until the cloud migration. And since CDA is just an interface the user wouldn't know the different.

I did notice one issue in your proposal. the delete endpoint suggests a "BEGIN" and "END" but the other endpoints contain no time information.

I would assume the intent would be to date every report with the date of the report and the date of generation.

If so, as far as the backend goes, it may be better to just use the text/binary timeseries as that what they would be and can take advantage of the existing versioned timeseries support. That doesn't exclude S3 as the current text/binary timeseries does use specially named entries. We're also considering moving those just columns on the respective timeseries. Probably worth doing some performance testing.

[krowvin]

the delete endpoint suggests a "BEGIN" and "END" but the other endpoints contain no time information.

Great point, I thought after the fact being able to delete a range maybe useful - but perhaps it's better to let users handle what to be deleted in the script itself and loop that given the other endpoints require/provide dates. (Begin/end would be optional)

If not just an internal timestamp that gets made when a report is actually uploaded creationTime or something

But text timeseries does also make sense.

I would think a reports endpoint would use BLOB/Clob/text-timeseries/etc - but expose it to endusers as a report centric endpoint

For example, author and agency do not exist unless you put them in the description on POST

I'm not set on any one means, but if we all used the same means it would make retrieving reports generically and if it's pointed at the same bucket WIll is using we have consistency across automated report uploads and manual reports.