Test database persistency on argocd

[anna-parker]

resolves #

preview URL:

Summary

Launch database then try adding more metadata fields to this persistent database, currently running here: https://test-presistent.loculus.org/.
Then:

• Confirm uploading sequences with the new fields does not crash the database:
• upload of new cchf sequences with empty lat and long fields is successful:

Hmmm... the page is behaving ok but I see some LAPIS issues when I look at argo (SILO pods are degraded):

[2024-09-18 08:33:54.882] [logger] [info] [logging_request_handler.cpp:21] Request Id [c562bba9-db79-40ef-aa8f-cb3dd0579a84] - Responding with status code 503
[2024-09-18 08:33:55.099] [logger] [info] [logging_request_handler.cpp:15] Request Id [ff72065f-bf61-46b8-91f0-bacb9bb41030] - Handling GET /info
[2024-09-18 08:33:55.099] [logger] [info] [error_request_handler.cpp:30] Caught exception: Database not initialized yet
[2024-09-18 08:33:55.099] [logger] [info] [logging_request_handler.cpp:21] Request Id [ff72065f-bf61-46b8-91f0-bacb9bb41030] - Responding with status code 503
[2024-09-18 08:33:55.343] [logger] [info] [database_directory_watcher.cpp:112] No data found in /data/ for ingestion

maybe this is just due to LAPIS being restarted? Hmm... search and download is working... interesting so LAPIS for cchf seems to have restarted logs look fine and downloads now include the new metadata fields, however for west nile and ebola where the SILO pods are degraded downloads do not include the new fields

Ok I tried to force the SILO pods to restart by deleting the old ones but they stay in this state.

So it would seem that until new sequences are uploaded which include all metadata fields the new SILO pods cannot start...

Screenshot

PR Checklist

• All necessary documentation has been adapted.
• The implemented feature is covered by an appropriate test.

[anna-parker]

Sadly I don't think this actually tested persistency as the database pod restarted after the last commit:

[theosanderson]

For info - this isn't consumed at all by Postgres so it could also just be removed

[theosanderson]

(It is used specifically to cause the db to be recreated because that is useful in some scenarios)

[anna-parker]

Looks like the culprit is this error in SILO preprocessing:

[2024-09-18 08:34:49.414] [logger] [error] [api.cpp:282] The field 'geoLocLatitude' which is contained in the database config is not contained in the input field '/preprocessing/input/data.ndjson.zst'.
std::exception

[anna-parker]

This makes sense - get-released-data just streams the contents of the processed metadata joint_metadata field in the sequence_entries_view. This metadata was computed by preprocessing before the additional metadata fields were added and thus does not contain the fields.

[anna-parker]

I think it makes total sense now that we see this behavior. We could either alter get-released-data to do this check (doesn't really make sense) or as already suggested just increase the version of the preprocessing pipeline causing all samples to have the new fields.

Also we allow the json to not include all metadata fields and just assume they are null if not given