Orion v2 docker setup

[Lezek123]

Depends on Joystream/orion#107

Can be tested by:

1. checking out Docker build adjustments orion#107
2. running make build-docker
3. checking out this branch
4. running export RUNTIME_PROFILE=TESTING && yarn build:packages && yarn start

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated (UTC)
pioneer-testnet	⬜️ Ignored (Inspect)		May 15, 2023 3:24pm

[mnaamani]

Ok I just tested this by incorrectly merging into my local ephesus branch. The orion-processor seems to be stopping/restaring:

2023-04-06 15:10:46 {"level":5,"time":1680779446699,"ns":"sqd:processor","err":{"generatedMessage":true,"code":"ERR_ASSERTION","actual":false,"expected":true,"operator":"==","stack":"AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:\n\n  (0, assert_1.default)(this.isV1000)\n\n    at MembersMemberRemarkedEvent.get asV1000 [as asV1000] (/squid/lib/types/events.js:554:30)\n    at processMemberRemarkedEvent (/squid/lib/mappings/membership/index.js:43:75)\n    at processEvent (/squid/lib/processor.js:171:11)\n    at /squid/lib/processor.js:184:23\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async TypeormDatabase.runTransaction (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:110:13)\n    at async TypeormDatabase.transact (/squid/node_modules/@subsquid/typeorm-store/lib/database.js:64:24)\n    at async Runner.process (/squid/node_modules/@subsquid/substrate-processor/lib/processor/runner.js:165:17)"}}

presumably the version of orion I built was not for the ephesus runtime. But does that indicate we need to deploy ephesus version of orion ahead of time like we are planning for the query-node?

The issue appears to be with handing MembersMemberRemarkedEvent

[mnaamani]

Works well with runtime from master branch.

[Lezek123]

presumably the version of orion I built was not for the ephesus runtime

If it was built from Joystream/orion#107 then it wasn't the Ephesus version, correct. The Ephesus version is in a separate PR: Joystream/orion#103

But does that indicate we need to deploy ephesus version of orion ahead of time like we are planning for the query-node?

Yes, we'll deploy the Ephesus version ahead of time (today).

The issue appears to be with handing MembersMemberRemarkedEvent

It's understandable, because the mappings on the current master branch assume the pre-ephesus version of MembersMemberRemarkedEvent. The distinction between different versions of this event is only supported on Joystream/orion#103

[mnaamani]

Since we have deployments running with postgres12, re-deploying v14 will cause a problem.

Here is an example output log for container where I started system with v12, then updated compose file to use v14, and simply did docker-compose up -d db so the recreated db using the same volume fails:

2023-05-15 12:22:25 2023-05-15 08:22:25.644 UTC [1] FATAL:  database files are incompatible with server
2023-05-15 12:22:25 2023-05-15 08:22:25.644 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 12, which is not compatible with this version 14.8 (Debian 14.8-1.pgdg110+1).
2023-05-15 12:22:12 
2023-05-15 12:22:12 PostgreSQL Database directory appears to contain a database; Skipping initialization
2023-05-15 12:22:12 
2023-05-15 12:22:25 
2023-05-15 12:22:25 PostgreSQL Database directory appears to contain a database; Skipping initialization
2023-05-15 12:22:25 
2023-05-15 12:22:51 
2023-05-15 12:22:51 PostgreSQL Database directory appears to contain a database; Skipping initialization
2023-05-15 12:22:51 
2023-05-15 12:22:51 2023-05-15 08:22:51.910 UTC [1] FATAL:  database files are incompatible with server

There is of course a tool available https://www.postgresql.org/docs/14/pgupgrade.html to transition.

Perhaps we can stick with v12 unless it is strictly required for some orion features, and we can provide a guide on how to migrate to v14?

[Lezek123]

Thank you very much for spotting this issue, I guess I didn't fully evaluate all the consequences of switching to v14.
The reason I did that was because I noticed that some SQL function used by one of the Orion queries was missing in v12 and I realized that we use v14 in the Orion repo. Switching to v14 in the monorepo as well fixed this particular issue.

I think the easiest solution would be to run 2 separate db services like db and orion-db, where db is v12 and used only by the query node, while orion-db would be v14 and would be only used by Orion.

[mnaamani]

Two database instances would work 👍
I'm sure there are additional benefits to updating production instances to newer version? Performance, security fixes, but we can have separate guides for query-node operators to do that manually.