bug(replicator): Replicator stops updating with many "foreign key constraint" errors #1556
Comments
github-actions
bot
added
the
s-triage
|
You can remove the foreign key constraint from the messages table and check whether it starts syncing. |
|
+1, i'm also having this same issue. |
|
This is likely due to the replicator missing certain hub events. #1553 should reduce the likelihood of this happening, but real fix is to retry the missing on chain event if this is detected. Will fix. |
|
+1, i'm also having this same issue. Any workaround you recommend until this gets fixed? I've tried pinning both my replicator and hub to 1.6.6 and syncing a fresh DB blasts my console output with these types of Errors edit 1: I flushed my local redis and restarted and now it seems to be syncing. This bug could be caused by the fact that i switched DBs and hub instances without clearing my redis? edit 2 ok now im having the same issue as OP |
|
any update on this? @sanjayprabhu |
|
@Bijan-Massoumi Are you still seeing this issue after upgrading to the latest version? |
|
@sanjayprabhu We're still seeing this issue (heavily) as of yesterday. Running hubble 1.9.8. |
|
@sanjayprabhu I'm also seeing this error pretty heavily at this point. Backfill is pretty much stuck at |
|
An update on the pinata side, we did a fresh start of a new hub / replicator and immediately removed the foreign key constraints. From what we're seeing:
@sanjayprabhu do you see any issues with this approach? |
|
Hey all, wanted to explain what we currently believe is happening here. A core assumption we made with the Replicator's design was that it should not process messages which refer to other messages/data that are missing. Hubs don't strictly follow this rule—a message can be valid even if it refers to another message that doesn't yet exist on the hub. Two simple examples of this are:
Similar situations apply for reactions, links, etc. Behavior is documented here, but we will call this out more clearly in the README: hub-monorepo/apps/replicator/src/migrations/001_initial_migration.ts Lines 38 to 43 in 0662959 The reason enforcing foreign key constraints are helpful in this context is that it can help find legitimate bugs (whether its due to missing FIDs from a lost event or something else)! However, for some purposes disabling them can be helpful in the short term, but I would not consider it a long-term solution—it's just masking the problem. We're giving a lot of thought to the current Replicator design. We hope to share more on the path forward soon. Another advantage to using foreign keys is that when you want to delete pruned/revoked/deleted messages, the constraints will automatically cascade the deletes with no additional application logic required by you. |
|
We now recommend you use the Shuttle package instead of the old replicator. |
What is the bug?
Replicator stopped updating last night (querying db missing data for last 15hours or so). Checking on the server the replicator process was logging many errors:
error: insert or update on table "messages" violates foreign key constraint "messages_signer_fid_foreign"How can it be reproduced? (optional)
Hub and Replicator both launched using the install scripts on the same Digital Ocean ubuntu 22.04 server about 3 days ago, so should have been version Hub Release 1.7.0 for both.
Hubble Dashboard wasn't showing anything unusual. Replicator redis WebUI (script defaults to port 9000) was showing ~300+ delayed jobs that all showed the same "foreign key constraint" error.
Not an urgent blocker as I was able to get it syncing again. => To attempt to clear up the issue I clicked "clean all" in the Delayed section of the Replicator Web UI. This didnt seem to fix it. I then brought hubble and replicator down via their scripts, and then brought them back up and at that point the replicator started syncing and appears to up to date again. Though the Replicator Web UI does still show ~40 of the "foreign key constraint" errors in the "Delayed" column.
Additional context (optional)
See Warpcast thread of myself(@cyrcus) and @haole discussing as we both ran into the error => https://warpcast.com/haole/0x89b9f44b
Screenshot of Replicator Redis UI:
See delayed job error logs below:
ProcessHubEvent Error
Data:
{ "data": { "hubEventJsonStr": "{\"type\":\"HUB_EVENT_TYPE_MERGE_MESSAGE\",\"id\":366630771179520,\"mergeMessageBody\":{\"message\":{\"data\":{\"type\":\"MESSAGE_TYPE_REACTION_ADD\",\"fid\":680,\"timestamp\":89486665,\"network\":\"FARCASTER_NETWORK_MAINNET\",\"reactionBody\":{\"type\":\"REACTION_TYPE_LIKE\",\"targetCastId\":{\"fid\":4163,\"hash\":\"VC+AAXFsvIxZVNzuq8l2vEIk6uE=\"}}},\"hash\":\"rQn98rxbgWI8JTK0fw+WvKoZLUE=\",\"hashScheme\":\"HASH_SCHEME_BLAKE3\",\"signature\":\"1yIVNQPNrgg4WEP+s6tvkXS3od8Aur80PngXZk7H4SdCWemSlKFo4SmFZoTJZe5JeUaAHMMHkhkFBRtC77D2Cg==\",\"signatureScheme\":\"SIGNATURE_SCHEME_ED25519\",\"signer\":\"sLPssdYagY3ui5gxFBDrfUr0nO3/Vay7mTrSXeESM4Q=\"},\"deletedMessages\":[]}}" }, "returnValue": null }Error:
The text was updated successfully, but these errors were encountered: