Encryption improvements #285

Half-Shot · 2020-11-23T17:08:07Z

Fixes #284

Depends on #295

Sometimes we get into a state where the bridge isn't reliably syncing messages from client's and so ends up in a broken state, this aims to fix that.

Half-Shot · 2021-01-15T16:24:17Z

src/bridge.ts

@@ -705,12 +706,16 @@ export class Bridge {
            const now = Date.now();
            for (const [key, entry] of this.intents.entries()) {
                // Do not delete intents that sync.
-                if (this.eeEventBroker?.shouldAvoidCull(entry.intent)) {


In the previous version of the code, we ended up never culling any Intents because of an erronous return statement (I'm good at those).

This change means that Intent jobs are cleaned up if they aren't used after INTENT_CULL_EVICT_AFTER_MS and the eeEventBroker is instantiated AND needs the user kept alive (e.g. they are the transport for encrypted messages in a room)

Half-Shot · 2021-01-15T16:25:07Z

src/components/encryption.ts

@@ -18,10 +20,12 @@ export interface ClientEncryptionSession {
    userId: string;
    deviceId: string;
    accessToken: string;
+    syncToken: string|null;


We now store the sync token to avoid doing an initial sync on startup, which was causing unbounded amounts of memory to be used.

Half-Shot · 2021-01-15T16:25:31Z

src/components/encryption.ts

@@ -107,7 +112,7 @@ export class EncryptedEventBroker {
    // We should probably make these LRUs eventually
    private eventsPendingAS: WeakEvent[] = [];

-    private syncingClients = new Set<any>();
+    private syncingClients = new Map<string, any>();


We now store the userId => client. The client from matrix-js-sdk doesn't have a type :(

Half-Shot · 2021-01-15T16:26:07Z

src/components/encryption.ts

@@ -142,6 +147,8 @@ export class EncryptedEventBroker {
        const existingUserForRoom = this.userForRoom.get(event.room_id);
        if (existingUserForRoom) {
            log.debug(`${existingUserForRoom} is listening for ${event.event_id}`);
+            // XXX: Sometimes the sync stops working, calling this will wake it up.
+            await this.startSyncingUser(existingUserForRoom);


This is unfortunate but sometimes even though a user is syncing for messages, they stop working. Calling this function checks that the user is syncing.

Half-Shot · 2021-01-15T16:26:15Z

src/components/encryption.ts

@@ -170,15 +177,18 @@ export class EncryptedEventBroker {

        const existingUser = membersForRoom.find((u) => [...this.userForRoom.values()].includes(u));
        if (existingUser) {
-            log.error(`${event.room_id} will be synced by ${existingUser}`);
+            log.debug(`${event.room_id} will be synced by ${existingUser}`);


Half-Shot · 2021-01-15T16:26:41Z

src/components/encryption.ts

@@ -261,20 +271,36 @@ export class EncryptedEventBroker {
        this.onEphemeralEvent(presenceEv);
    }

+    private async onSync(userId: string, state: string, data: {nextSyncToken: string; catchingUp: boolean;}) {


Update the stored sync token on insert

Half-Shot · 2021-01-15T16:29:30Z

src/components/encryption.ts

    /**
     * Start a sync loop for a given bridge user
     * @param userId The user whos matrix client should start syncing
     */
-    public async startSyncingUser(userId: string) {
-        log.info(`Starting to sync ${userId}`);
+    public async startSyncingUser(userId: string): Promise<void> {


This function was refactored to:

Check if a client is already syncing

Check if a client has stopped syncing and needs restarting

Half-Shot · 2021-01-15T16:31:03Z

src/components/intent.ts

-            // We *need* to sync before we can send a message.
-            await this.ensureRegistered();
-            await this.encryption.ensureClientSyncingCallback();
+            let encrypted = false;


We previously started syncing any client that sent a message to a room, but that's inefficent if the room is unencrypted as we do not need to sync.

This code checks to see if a room is encrypted.

Half-Shot · 2021-01-15T16:33:28Z

src/components/intent.ts

@@ -767,6 +803,9 @@ export class Intent {
        else if (event.type === "m.room.power_levels") {
            this._powerLevels[event.room_id] = event.content as unknown as PowerLevelContent;
        }
+        else if (event.type === "m.room.encryption") {


Ensure we catch any rooms that get encryption turned on :)

Half-Shot · 2021-01-15T16:33:55Z

src/components/prometheusmetrics.ts

@@ -380,11 +380,11 @@ export class PrometheusMetrics {
            // TODO: Ideally these metrics would be on a different port.
            // For now, leave this unauthenticated.
            checkToken: false,
-            handler: (_req: Request, res: Response) => {
+            handler: async (_req: Request, res: Response) => {


This appears in #296 but I needed it to work for my encrypted bridge deployment #sorry

No need to be sorry. Great work!

jaller94

Blindly trusting ev.algorithm to be a string is a bit to trusting from my perspective.
Everything else looks good.

src/components/encryption.ts

src/components/intent.ts

jaller94 · 2021-01-22T10:39:40Z

src/components/prometheusmetrics.ts

@@ -380,11 +380,11 @@ export class PrometheusMetrics {
            // TODO: Ideally these metrics would be on a different port.
            // For now, leave this unauthenticated.
            checkToken: false,
-            handler: (_req: Request, res: Response) => {
+            handler: async (_req: Request, res: Response) => {


No need to be sorry. Great work!

Co-authored-by: Christian Paul <christianp@matrix.org>

Half-Shot and others added 3 commits January 14, 2021 12:50

Filter sync properly

a390458

fix ephemeral

d63f47c

Get the correct sync state

da5f57b

Half-Shot force-pushed the hs/fix-sync-fails branch from df23a79 to da5f57b Compare January 14, 2021 14:08

Half-Shot added 2 commits January 14, 2021 14:09

changelog

5803d52

Ensure we stop clients when we cull them

edd7668

Half-Shot force-pushed the hs/fix-sync-fails branch from f745f5b to edd7668 Compare January 14, 2021 14:24

Half-Shot added 5 commits January 14, 2021 17:59

More supporting code to evict the right syncers at the right time

e7c28c3

Support sync tokens

5fe859f

Metrics are now async

b769e8c

Check if a room is encrypted before syncing

f2d091c

fix cache

c563127

Half-Shot changed the title ~~Filter sync properly~~ Encryption improvements Jan 15, 2021

Half-Shot changed the base branch from develop to 265-add-contributing-docs January 15, 2021 16:22

Half-Shot changed the base branch from 265-add-contributing-docs to develop January 15, 2021 16:22

Half-Shot commented Jan 15, 2021

View reviewed changes

Remove stat

09bf392

Half-Shot commented Jan 15, 2021

View reviewed changes

Add comments

2643d81

Half-Shot commented Jan 15, 2021

View reviewed changes

Half-Shot added 2 commits January 15, 2021 16:31

comments

a12cb7c

ensure we only operate on state events

ee87eba

Half-Shot commented Jan 15, 2021

View reviewed changes

fix lint

729a604

Half-Shot mentioned this pull request Jan 15, 2021

Add supporting code for latest enc bridge efforts matrix-org/matrix-appservice-slack#565

Merged

Half-Shot requested a review from a team January 15, 2021 17:33

jaller94 suggested changes Jan 22, 2021

View reviewed changes

Half-Shot and others added 3 commits January 22, 2021 11:07

Ensure types for encrypted rooms

e8f6816

Co-authored-by: Christian Paul <christianp@matrix.org>

Add parenthesis

a7ca345

fix types

acde98b

Half-Shot requested a review from jaller94 January 22, 2021 12:23

Update intent.ts

c87269f

Half-Shot merged commit 43b9f24 into develop Jan 22, 2021

jaller94 deleted the hs/fix-sync-fails branch April 26, 2021 13:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encryption improvements #285

Encryption improvements #285

Half-Shot commented Nov 23, 2020 •

edited

Loading

Half-Shot Jan 15, 2021

Half-Shot Jan 15, 2021

Half-Shot Jan 15, 2021

Half-Shot Jan 15, 2021

Half-Shot Jan 15, 2021

Half-Shot Jan 15, 2021

Half-Shot Jan 15, 2021

Half-Shot Jan 15, 2021

Half-Shot Jan 15, 2021

Half-Shot Jan 15, 2021

jaller94 Jan 22, 2021

jaller94 left a comment

jaller94 Jan 22, 2021

Encryption improvements #285

Encryption improvements #285

Conversation

Half-Shot commented Nov 23, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaller94 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Half-Shot commented Nov 23, 2020 •

edited

Loading