Something odd happened to my device list/keys, and then my messages wouldn't federate anymore.

[lampholder]

lant.uk synapse was accepting messages, but they weren't making it to matrix.org.
Anecdotally this seemed to start at the same time I logged out of Riot Android to test another account (not on lant.uk), then tried to log in to my main account again. This login attempt never succeeded.

Synapse was logging the following:

Mar 28 00:11:11 thor matrix-synapse[31968]: 2020-03-27 23:11:11,815 - synapse.federation.sender.per_destination_queue - 332 - ERROR - federation_transaction_transmission_loop-0 - TX [matrix.org] Failed to send transaction
Mar 28 00:11:11 thor matrix-synapse[31968]: Traceback (most recent call last):
Mar 28 00:11:11 thor matrix-synapse[31968]:   File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
Mar 28 00:11:11 thor matrix-synapse[31968]:     result = g.send(result)
Mar 28 00:11:11 thor matrix-synapse[31968]: StopIteration: 0
Mar 28 00:11:11 thor matrix-synapse[31968]: During handling of the above exception, another exception occurred:
Mar 28 00:11:11 thor matrix-synapse[31968]: Traceback (most recent call last):
Mar 28 00:11:11 thor matrix-synapse[31968]:   File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
Mar 28 00:11:11 thor matrix-synapse[31968]:     result = g.send(result)
Mar 28 00:11:11 thor matrix-synapse[31968]: StopIteration: 0
Mar 28 00:11:11 thor matrix-synapse[31968]: During handling of the above exception, another exception occurred:
Mar 28 00:11:11 thor matrix-synapse[31968]: Traceback (most recent call last):
Mar 28 00:11:11 thor matrix-synapse[31968]:   File "/usr/local/lib/python3.7/site-packages/synapse/federation/sender/per_destination_queue.py", line 201, in _transaction_transmission_loop
Mar 28 00:11:11 thor matrix-synapse[31968]:     limit
Mar 28 00:11:11 thor matrix-synapse[31968]:   File "/usr/local/lib/python3.7/site-packages/synapse/federation/sender/per_destination_queue.py", line 368, in _get_device_update_edus
Mar 28 00:11:11 thor matrix-synapse[31968]:     self._destination, last_device_list, limit=limit
Mar 28 00:11:11 thor matrix-synapse[31968]:   File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
Mar 28 00:11:11 thor matrix-synapse[31968]:     result = result.throwExceptionIntoGenerator(g)
Mar 28 00:11:11 thor matrix-synapse[31968]:   File "/usr/local/lib/python3.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
Mar 28 00:11:11 thor matrix-synapse[31968]:     return g.throw(self.type, self.value, self.tb)
Mar 28 00:11:11 thor matrix-synapse[31968]:   File "/usr/local/lib/python3.7/site-packages/synapse/storage/data_stores/main/devices.py", line 233, in get_device_updates_by_remote
Mar 28 00:11:11 thor matrix-synapse[31968]:     destination, from_stream_id, query_map
Mar 28 00:11:11 thor matrix-synapse[31968]:   File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
Mar 28 00:11:11 thor matrix-synapse[31968]:     result = g.send(result)
Mar 28 00:11:11 thor matrix-synapse[31968]:   File "/usr/local/lib/python3.7/site-packages/synapse/storage/data_stores/main/devices.py", line 325, in _get_device_update_edus_by_remote
Mar 28 00:11:11 thor matrix-synapse[31968]:     result["keys"].setdefault("signatures", {}).setdefault(
Mar 28 00:11:11 thor matrix-synapse[31968]: KeyError: 'keys'

My db state was this:

synapse=# select * from devices where user_id ='@tom:lant.uk' and hidden = False;
   user_id    |    device_id     |         display_name          |   last_seen   |       ip       |                                                                           user_agent                                                                            | hidden
--------------+------------------+-------------------------------+---------------+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+--------
 @tom:lant.uk | Mautrix-Facebook | Generic Matrix Client · Linux | 1585351781327 | -              | Python/3.8 aiohttp/3.6.2                                                                                                                                        | f
 @tom:lant.uk | ABICGMVRNF       | Riot · Desktop · macOS        | 1584985239789 | XXXXXXXX | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) RiotNightly/2020032301 Chrome/80.0.3987.134 Electron/8.0.3 Safari/537.36 | f
 @tom:lant.uk | HWVQMHGYDM       | Riot · Firefox · macOS        | 1585351825192 | XXXXXXXXX | Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:76.0) Gecko/20100101 Firefox/76.0                                                                              | f
 @tom:lant.uk | EYDLDOKXIK       |                               |               |                |                                                                                                                                                                 | f
 @tom:lant.uk | BKOEIBIDJW       | Mobile                        | 1585349903283 | XXXXXXXX | Riot.im/0.9.9-dev (Linux; U; Android 9; CLT-L09 Build/HUAWEICLT-L0; Flavour GooglePlay; MatrixAndroidSDK 0.9.31-dev)                                            | f
(5 rows)

synapse=# select * from e2e_device_keys_json where user_id = '@tom:lant.uk';;
   user_id    | device_id  |  ts_added_ms  |                                                                                                                                                                                                         key_json
--------------+------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 @tom:lant.uk | ABICGMVRNF | 1584632687735 | {"algorithms":["m.olm.v1.curve25519-aes-sha2","m.megolm.v1.aes-sha2"],"device_id":"ABICGMVRNF","keys":{LIST_OF_KEYS},"signatures":{"@tom:lant.uk":{SIGNATURE_STUFF}},"user_id":"@tom:lant.uk"}
 @tom:lant.uk | HWVQMHGYDM | 1584973633940 | {"algorithms":["m.olm.v1.curve25519-aes-sha2","m.megolm.v1.aes-sha2"],"device_id":"HWVQMHGYDM","keys":{LIST_OF_KEYS},"signatures":{"@tom:lant.uk":{SIGNATURE_STUFF}},"user_id":"@tom:lant.uk"}
(2 rows)

I guess the stuff in e2e_device_keys_json isn't sensitive but have redacted anyway. I have a copy if it's useful.

[lampholder]

I restored outbound federation by logging out EYDLDOKXIK and BKOEIBIDJW using the Riot Web UI.

[Half-Shot]

Looks like it's a fairly common occurrence in https://sentry.matrix.org/sentry/synapse-matrixorg/issues/73896/events/. And it's massively ramped up in occurances since March 26th.

[anoadragon453]

This is suspected to be caused by a client uploading signatures of their device keys, but failing to upload the keys themselves, and then some Synapse code handling that situation badly.

The Synapse code is fixed in v1.12.1 (of which there is currently an RC for. Unfortunately, if you're affected by this bug, other servers will need to upgrade before they can continue to send your server events.

Deleting devices may or may not be a valid workaround at this time. The best suggestion is to wait for the full v1.12.1 release and then to urge others to upgrade. This bug was introduced in v1.11.0, so servers on a version before that will not be affected.

[anoadragon453]

Closing for now, if you still experience this behaviour on v1.12.1, please open a new issue.