Skip to content

Fix Timeseries/recent endpoint speed #1183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 9 commits into
base: develop
Choose a base branch
from

Conversation

rohaan-rma
Copy link
Contributor

@rohaan-rma rohaan-rma commented Jul 10, 2025

Proposed query update that changes a single Timeseries/Recent request from 30 seconds down to half a second on our SWT db export.

WITH base_ids AS (
    SELECT DISTINCT
        ( ts_code ),
        cwms_ts_id
    FROM
        av_cwms_ts_id2
    WHERE
        cwms_ts_id IN ( 'HAMM.Stage-Alt.Inst.15Minutes.0.Decodes-Raw', 'HAMM.Stage.Inst.15Minutes.0.Decodes-Raw' )
), av_tsv_limited AS (
    SELECT
        *
    FROM
        at_tsv_2023
    WHERE
        date_time BETWEEN DATE '2024-06-24' AND DATE '2024-07-22'
        AND ts_code IN (
            SELECT
                ts_code
            FROM
                base_ids
        )
    UNION ALL
    SELECT
        *
    FROM
        at_tsv_2024
    WHERE
        date_time BETWEEN DATE '2024-06-24' AND DATE '2024-07-22'
        AND ts_code IN (
            SELECT
                ts_code
            FROM
                base_ids
        )
), max_values AS (
    SELECT
        b.cwms_ts_id,
        t.ts_code,
        t.date_time,
        t.value,
        t.version_date,
        t.data_entry_date,
        t.quality_code,
        ex.earliest_entry_time       start_date,
        ex.latest_entry_time         end_date,
        MAX(t.date_time)
        OVER(PARTITION BY t.ts_code) AS max_date_time
    FROM
             av_tsv_limited t
        JOIN base_ids      b ON b.ts_code = t.ts_code
        JOIN at_ts_extents ex ON ex.ts_code = t.ts_code
                                 AND ex.version_time = t.version_date
)
SELECT
    max_values.cwms_ts_id,
    max_values.date_time,
    max_values.version_date,
    max_values.data_entry_date,
    max_values.quality_code,
    max_values.start_date,
    max_values.end_date,
    cwms_util.get_default_units(
        cwms_ts.get_base_parameter_id(max_values.ts_code),
        'EN'
    )                    AS def_units,
    max_values.date_time AS max_date_time,
    cwms_util.convert_units(max_values.value,
                            cwms_util.get_default_units(
                      cwms_ts.get_base_parameter_id(max_values.ts_code),
                      'SI'
                  ),
                            cwms_util.get_default_units(
                      cwms_ts.get_base_parameter_id(max_values.ts_code),
                      'EN'
                  ))                   AS value_at_max_date
FROM
    max_values
WHERE
    max_values.date_time = max_values.max_date_time;

Copy link
Contributor

@MikeNeilson MikeNeilson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable. Definitely need to chew on it a bit. Fairly complex query, though I doubt unnecessarily so.

.from(baseIds);

// references to appropriate year tables
Table<?> AT_TSV_2023_TABLE = table(name("AT_TSV_2023"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should be AT_TSV_PREV_YEAR and AT_TSV_CURRENT_YEAR and calculated at usage. Otherwise we'll have to remember to update them constantly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup you're correct. I just had those as place holders for now.

@adamkorynta adamkorynta changed the title Fix Timeseries endpoint speed Fix Timeseries/recent endpoint speed Jul 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

timeseries/recent endpoint is much slower than the /timeseries endpoint
2 participants