🔨 store grapher configs in R2 when edited in the admin

[danyx23]

This PR does 2 things at the high level: it stores grapher configs in R2 when they are edited in the admin, and it adds a tool for syncing the content of the chart_configs table into R2 (adding/updating missing ones and deleting superfluous ones).

Charts are stored in R2 in two folders for now: one that is used for all configs addressed by chart UUID, and one addressed by slug for only published standalone charts.

This PR adds a hash of the full config in the chart_configs table. I started out using a SHA-1 and was happy that R2 supports those in addition to MD5, only to discover later on when I wrote the sync tool that the S3 API list operation has no support for hashes other than MD5. Since the main point of the hash is facilitate efficient comparison between set of configs in the DB and those in R2 I then rewrote the hash to be md5. The hash is stored in base64 encoding since this is also what is used and returned in most api calls (the only exception is the ETAG where a hex serialization in double quotes is used 🤷)

We already have a key configured for interacting with R2 so I renamed the settings related to this key to have a generic (non-image specific) name.

To test this:

• set the R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY to a personal CF R2 token that you generate. If you already did this previously for images (i.e. if your .env file already has IMAGE_HOSTING_R2_ACCESS_KEY_ID) then rename these two keys AND make sure that your personal key has access to the owid-grapher-configs-staging bucket (CF dashboard -> R2 -> manage api tokens in the top right)
• set GRAPHER_CONFIG_R2_BUCKET to owid-grapher-configs-staging and GRAPHER_CONFIG_R2_BUCKET_PATH to devs/YOURNAME

Then when you interact with the admin and save graphers, you should see these charts show up as json files in the R2 bucket in the owid-grapher-configs-staging bucket.

[danyx23]

• 🔨 store grapher configs in R2 when edited in the admin #3827 : 2 dependent PRs (#3847 , #3867 ) 👈
• 🎉 (grapher) make charts inherit indicator-level settings / TAS-567 #3793 : 1 other dependent PR (#3886 )
• 🔨 (grapher) let chart configs inherit default values / TAS-565 #3782 : 1 other dependent PR (#3890 )
• 🔨 (db) add chart_configs table / TAS-563 #3767 : 1 other dependent PR (#3743 )
• 🔨 Add wrangler.toml file #3846
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @danyx23 and the rest of your teammates on Graphite

[owidbot]

Quick links (staging server):

Site	Admin	Wizard

Login: ssh owid@staging-site-grapher-configs-in-r2

• Site-screenshots: https://github.com/owid/site-screenshots/compare/grapher-configs-in-r2
• SVG tester: https://github.com/owid/owid-grapher-svgs/compare/grapher-configs-in-r2

SVG tester:

Number of differences (default views): 35 (9a4c56) ❌
Number of differences (all views): 0 ✅

Edited: 2024-09-09 12:02:51 UTC
Execution time: 1.20 seconds

[danyx23]

Do we have a good sense of how we can reconstruct chart revisions if we're only serializing the patch?

Without inheritance it's the same as before but with inheritance it's true that this becomes a bit more tricky. Let's chat about it a bit today. I think storing the full config wouldn't be better though.

[danyx23]

Merge activity

• Sep 9, 10:35 AM EDT: @danyx23 started a stack merge that includes this pull request via Graphite.
• Sep 9, 10:35 AM EDT: @danyx23 merged this pull request with Graphite.