Is there a way to get all data?

[andyvan-trabus]

I'm trying to get historical inflow/outflow data. If I use the Time Series tab at the following URL (WM Data Dissemination site) https://water.usace.army.mil/a2w/f?p=100:1::::::#, I'm able to get a graph that goes from December 6, 2020 to December 6, 2022.

When I use the Swagger UI to get timeseries metadata for Joe Pool Lake ("JPLT2.Flow.") and look at the extents, I can find several that go back to 2020-12-03 or 2020-12-02, but none of them, according to the metadata, provide data any newer than 2022-05-10.

Is there a way to go back older? Is there a way to get recent flow data for Joe Pool?

Request
https://cwms-data.usace.army.mil/cwms-data/catalog/TIMESERIES?office=SWF&like=JPLT2.%2AFlow.%2A

Response
{
"total": 19,
"entries": [
{
"office": "SWF",
"name": "JPLT2.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI",
"units": "cms",
"interval": "1Hour",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [
{
"earliest-time": "2020-12-03T12:00:00Z",
"last-update": "2022-05-08T08:42:23.645258Z",
"latest-time": "2022-05-08T08:00:00Z"
}
],
"cursor": "SWF/JPLT2.FLOW-OUT.INST.1HOUR.0.REV-SWF-REGI"
},
{
"office": "SWF",
"name": "JPLT2-Spillway.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI",
"units": "cms",
"interval": "1Hour",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [
{
"earliest-time": "2020-12-03T12:00:00Z",
"last-update": "2022-05-08T08:42:33.159419Z",
"latest-time": "2022-05-08T08:00:00Z"
}
],
"cursor": "SWF/JPLT2-SPILLWAY.FLOW-OUT.INST.1HOUR.0.REV-SWF-REGI"
},
{
"office": "SWF",
"name": "JPLT2.Flow-In.Ave.~1Day.1Day.Rev-SWF-REGI",
"units": "cms",
"interval": "~1Day",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [
{
"earliest-time": "2020-12-02T06:00:00Z",
"last-update": "2022-05-08T12:07:56.303948Z",
"latest-time": "2022-02-01T06:00:00Z"
}
],
"cursor": "SWF/JPLT2.FLOW-IN.AVE.~1DAY.1DAY.REV-SWF-REGI"
},
{
"office": "SWF",
"name": "JPLT2.Flow-In.Ave.~1Day.1Day.Computed-SWF-REGI",
"units": "cms",
"interval": "~1Day",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [
{
"earliest-time": "2020-12-01T06:00:00Z",
"last-update": "2022-05-10T12:09:37.171636Z",
"latest-time": "2022-05-10T05:00:00Z"
}
],
"cursor": "SWF/JPLT2.FLOW-IN.AVE.~1DAY.1DAY.COMPUTED-SWF-REGI"
},
{
"office": "SWF",
"name": "JPLT2-Leakage.Flow-Out.Ave.~1Day.1Day.Rev-SWF-REGI",
"units": "cms",
"interval": "~1Day",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [
{
"earliest-time": "2021-07-31T05:00:00Z",
"last-update": "2022-05-10T23:46:45.38643Z",
"latest-time": "2022-05-10T05:00:00Z"
}
],
"cursor": "SWF/JPLT2-LEAKAGE.FLOW-OUT.AVE.~1DAY.1DAY.REV-SWF-REGI"
},
{
"office": "SWF",
"name": "JPLT2-Pump.Flow-Out.Ave.~1Day.1Day.Rev-SWF-REGI",
"units": "cms",
"interval": "~1Day",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [
{
"earliest-time": "2021-09-29T05:00:00Z",
"last-update": "2022-05-08T23:49:54.047311Z",
"latest-time": "2022-05-08T05:00:00Z"
}
],
"cursor": "SWF/JPLT2-PUMP.FLOW-OUT.AVE.~1DAY.1DAY.REV-SWF-REGI"
},
{
"office": "SWF",
"name": "JPLT2-Gated_Total.Flow-Out.Ave.~1Day.1Day.Rev-SWF-REGI",
"units": "cms",
"interval": "~1Day",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [
{
"earliest-time": "2021-07-31T05:00:00Z",
"last-update": "2022-05-09T01:25:37.285713Z",
"latest-time": "2022-05-08T05:00:00Z"
}
],
"cursor": "SWF/JPLT2-GATED_TOTAL.FLOW-OUT.AVE.~1DAY.1DAY.REV-SWF-REGI"
},
{
"office": "SWF",
"name": "JPLT2-Spillway.Flow-Out.Ave.~1Day.1Day.Rev-SWF-REGI",
"units": "cms",
"interval": "~1Day",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [
{
"earliest-time": "2022-02-02T06:00:00Z",
"last-update": "2022-05-09T01:25:37.903892Z",
"latest-time": "2022-05-08T05:00:00Z"
}
],
"cursor": "SWF/JPLT2-SPILLWAY.FLOW-OUT.AVE.~1DAY.1DAY.REV-SWF-REGI"
},
{
"office": "SWF",
"name": "JPLT2.Flow-Out.Ave.~1Day.1Day.Rev-SWF-REGI",
"units": "cms",
"interval": "~1Day",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [
{
"earliest-time": "2020-08-31T05:00:00Z",
"last-update": "2022-05-09T01:25:46.953201Z",
"latest-time": "2022-05-08T05:00:00Z"
}
],
"cursor": "SWF/JPLT2.FLOW-OUT.AVE.~1DAY.1DAY.REV-SWF-REGI"
},
{
"office": "SWF",
"name": "JPLT2.Flow.Inst.1Hour.0.Decodes-Rev",
"units": "cms",
"interval": "1Hour",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [],
"cursor": "SWF/JPLT2.FLOW.INST.1HOUR.0.DECODES-REV"
},
{
"office": "SWF",
"name": "JPLT2-LF1.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI",
"units": "cms",
"interval": "1Hour",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [],
"cursor": "SWF/JPLT2-LF1.FLOW-OUT.INST.1HOUR.0.REV-SWF-REGI"
},
{
"office": "SWF",
"name": "JPLT2-Gated_Total.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI",
"units": "cms",
"interval": "1Hour",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [],
"cursor": "SWF/JPLT2-GATED_TOTAL.FLOW-OUT.INST.1HOUR.0.REV-SWF-REGI"
},
{
"office": "SWF",
"name": "JPLT2.Flow-In.Inst.1Hour.0.Decodes-Comp",
"units": "cms",
"interval": "1Hour",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [],
"cursor": "SWF/JPLT2.FLOW-IN.INST.1HOUR.0.DECODES-COMP"
},
{
"office": "SWF",
"name": "JPLT2-LF1.Opening-Low_Flow_Gate.Inst.0.0.Regi-Manual",
"units": "m",
"interval": "0",
"interval-offset": -2147483648,
"time-zone": "US/Central",
"extents": [],
"cursor": "SWF/JPLT2-LF1.OPENING-LOW_FLOW_GATE.INST.0.0.REGI-MANUAL"
},
{
"office": "SWF",
"name": "JPLT2.Flow-Difference-Stor.Ave.~1Day.1Day.Rev-SWF-Comp",
"units": "cms",
"interval": "~1Day",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [],
"cursor": "SWF/JPLT2.FLOW-DIFFERENCE-STOR.AVE.~1DAY.1DAY.REV-SWF-COMP"
},
{
"office": "SWF",
"name": "JPLT2-Leakage.Flow-Out.Inst.1Hour.0.Rev-SWF-REGI",
"units": "cms",
"interval": "1Hour",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [],
"cursor": "SWF/JPLT2-LEAKAGE.FLOW-OUT.INST.1HOUR.0.REV-SWF-REGI"
},
{
"office": "SWF",
"name": "JPLT2.Flow-Evap.Ave.~1Day.1Day.Rev-SWF-REGI",
"units": "cms",
"interval": "~1Day",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [],
"cursor": "SWF/JPLT2.FLOW-EVAP.AVE.~1DAY.1DAY.REV-SWF-REGI"
},
{
"office": "SWF",
"name": "JPLT2.Flow-In.Inst.1Hour.0.DSS-Math-Comp",
"units": "cms",
"interval": "1Hour",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [],
"cursor": "SWF/JPLT2.FLOW-IN.INST.1HOUR.0.DSS-MATH-COMP"
},
{
"office": "SWF",
"name": "JPLT2.Flow-Out.Inst.1Hour.0.Fcst-SWF-CWMS",
"units": "cms",
"interval": "1Hour",
"interval-offset": 0,
"time-zone": "US/Central",
"extents": [],
"cursor": "SWF/JPLT2.FLOW-OUT.INST.1HOUR.0.FCST-SWF-CWMS"
}
],
"page-size": 500
}

[andyvan-trabus]

Here's my screenshot showing that I could get data from December 6, 2020 to December 6, 2020 from the WM Data Dissemination site.

Note the red text that says I should "use dataservices for larger timeframe requests." The Swagger UI at https://cwms-data.usace.army.mil/cwms-data/swagger-ui.html is what I'm supposed to be using for larger timeframe requests, right?

[MikeNeilson]

Yes, or after you've configured everything in the swagger-ui and it works there should be a curl command line presented, you can use that directly in the future.

However 2 things.
1: not all of the historical data has actually been pushed to our central data service. You'll have to contact the District to get the period of record
2: There are some configuration issues we're working out so even with the pagination it's difficult to get more than a week of data at a time right now. Someone is looking into that though, unfortunately we don't have an ETA at this time for when that will be resolved.

While this services is supposed to be available for what you're trying to do it sounds like you need a lot of historical data, you may be better served using it to determine which time series to ask for and getting the historical from the district.