-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
My HS has accumulated thousands of unreferenced state groups #3364
Comments
worse, when I go to purge history, the unreferenced state groups start turning into non-delta state groups, which makes the whole thing worse. |
redundant state groups:
(empirically most of them seem to be coming from HQ) |
I seem to have them as well. I created that table on my system as well and get the following response:
Just to note: I have not run any purge commands yet |
it's easier to create the new state group as a delta from the existing one. (There's an outside chance this will help with #3364)
#3625 might be related |
neil I don't think this can be a p2; it's a real blocker on cleaning up disk space |
Another occasion that (I think) this happens is when we have a fork in the DAG, with different state on the two sides of the fork, and the next event (which heals the fork) is itself another state event. We create a new state group when we state-resolve the two sides for the fork (which is important for caching state res), but that SG is never actually (directly) used because we then create another SG to include the updated state. |
We have kind of a big disk filling database too ~(45G) and ~40 Users. We started to purge the history some time ago, monthly, so the db should contain only the data of the last 365 days with a sloop of 30 days. So I was curious have many tuples in
Which resulted in 1.388.475 affected tupels, which is kind of nothing in contrast to 84.141.600 tupels in the table. So this Is definitely a thing, but my guess is that we have other waste in that database, or is this a "normal/to be expected" size? |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Those 1.3M tuples will just be the deltas from the previous state groups - probably only one or two rows per state group. The problem comes when a state group is removed, which means that any other state group which references it will have to be converted from delta storage to a absolutes - ie, we will have to store every single state event for the room for each of those state groups. Suppose we have three state groups in a room, 1, 2, and 3. 1 is the first state group, and 2 and 3 are both stored as deltas from 1:
SG1 and SG3 are both used for a number of events in the room, but as per this bug, SG2 is unused. Now we purge some events from this room. SG1 and SG3 are detected as unused and deleted. However, SG2 is losing its parent, so needs "de-deltaing". Multiply this effect by 1.3M, and you have a real problem. |
hi, i believe i'm facing the same problem described a year earlier in this issue: the whole database weigh 14 GB (7 users registered only, no huge rooms joined...)
here are the biggest tables:
isn't there something to do? it's labeled P1 and i think truly critical. -- edit 10 days later |
Coming back to this topic. I just run the queries from above and they still find unreferenced state groups (~10k), since my knowledge about the database schema is kind of nearly none existence, can you please provide us with a query to safely getting rid of those rows. |
This comment has been minimized.
This comment has been minimized.
(Sidenote: irc bridged rooms are far beyond MatrixHQ now, with |
Another factor in this is that, as of #6320, we now create a new state group for any new state event which is submitted via the C-S API, even if that event is not accepted. |
The long and the short of this is that I think we need a script which will gradually walk the state_groups table, looking for redundant state groups and removing them. (it would also be nice to stop some of the state groups being created in the first place, but that's a bit harder.) |
cactus.chat is heavily affected by this, so it is probably caused by bots/bridges/appservices. Our homeserver is strange, because there are no users, only guests and mostly room.membership events.
|
Guests count as users |
I have about 1.5 million unreferenced state groups right now, is there a recommended way to deal with them? |
The recommended way to remove unreferenced state groups is via https://github.com/erikjohnston/synapse-find-unreferenced-state-groups |
The README for that tool still says "Do not blindly delete all the state groups that are returned by this tool" though. |
Indeed. Shut down synapse first. Or omit the last, say, 100 results from that tool. I didn't say it was a good solution to the problem. Just that it's a way to deal with it. |
So, as long as synapse is not running during the whole cleanup process, the output of the tool can be used blindly? |
Yes. |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
Once again: please keep the conversation on topic. General grumbling about the size of the database is off-topic; as is anything that is improved by https://github.com/matrix-org/rust-synapse-compress-state. This issue is specifically about an accumulation of unreferenced state groups. |
Could you please confirm that, as of Synapse |
... which are filling up my disk :(
To check if you are also affected, run this query:
if you see numbers in the thousands, then it is this issue. Otherwise, you're not affected by this issue.
The text was updated successfully, but these errors were encountered: