From 0db8654360d88d06856ea538b9e0df0e50998df7 Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Wed, 6 May 2020 22:25:08 +0100 Subject: [PATCH 1/8] Redis configuration details, draft --- docs/workers.md | 65 ++++++++++++++++++++++++++++++++----------------- 1 file changed, 43 insertions(+), 22 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index cc0b23197ffd..dce82b0e0793 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -1,23 +1,37 @@ # Scaling synapse via workers -Synapse has experimental support for splitting out functionality into -multiple separate python processes, helping greatly with scalability. These +For small instances it recommended to run Synapse in monolith mode (the +default). For larger instances where performance is a concern it can be helpful +to split out functionality into multiple separate python processes. These processes are called 'workers', and are (eventually) intended to scale horizontally independently. -All of the below is highly experimental and subject to change as Synapse evolves, -but documenting it here to help folks needing highly scalable Synapses similar -to the one running matrix.org! +Synapse's worker support is under active development and subject to change as +we attempt to rapidly scale ever larger Synapse instances. However we are +documenting it here to help admins needing a highly scalable Synapse instance +similar to the one running matrix.org. -All processes continue to share the same database instance, and as such, workers -only work with postgres based synapse deployments (sharing a single sqlite -across multiple processes is a recipe for disaster, plus you should be using -postgres anyway if you care about scalability). +All processes continue to share the same database instance, and as such, +workers only work with postgres based synapse deployments. SQlite should only +be used for demo purposes and any admin considering workers should already by +running postgres. + +Originally the workers communicated with the master synapse process via a +synapse-specific TCP protocol called 'replication' - analogous to MySQL or +Postgres style database replication; feeding a stream of relevant data to the +workers so they can be kept in sync with the main synapse process and database +state. + +More recently (v1.13.0) Redis support was added, Redis is an alternative to +direct tcp connections to the master: rather than all the workers connecting to +the master, all the workers and the master connect to redis, which relays +replication commands between processes. This can give a significant cpu saving +on the master and will be a prerequisite for upcoming performance improvements. + +This doc covers details of both approaches (TCP Replication and Redis +Replication), however the TCP Replication implementation will be deprecated in +favour of Redis. -The workers communicate with the master synapse process via a synapse-specific -TCP protocol called 'replication' - analogous to MySQL or Postgres style -database replication; feeding a stream of relevant data to the workers so they -can be kept in sync with the main synapse process and database state. ## Configuration @@ -27,9 +41,18 @@ the correct worker, or to the main synapse instance. Note that this includes requests made to the federation port. See [reverse_proxy.md](reverse_proxy.md) for information on setting up a reverse proxy. -To enable workers, you need to add two replication listeners to the master -synapse, e.g.: +To enable workers, you need to add replication listeners to the master +synapse, in the case of Redis Replication you need only configure an HTTP listener e.g.: + + listeners: + # The HTTP replication port + - port: 9093 + bind_address: '127.0.0.1' + type: http + resources: + - names: [replication] +For TCP Replication listeners: # The TCP replication port - port: 9092 @@ -42,13 +65,10 @@ synapse, e.g.: resources: - names: [replication] -Under **no circumstances** should these replication API listeners be exposed to -the public internet; it currently implements no authentication whatsoever and is -unencrypted. -(Roughly, the TCP port is used for streaming data from the master to the -workers, and the HTTP port for the workers to send data to the main -synapse process.) +Under **no circumstances** should replication API listeners be exposed to +the public internet; they currently do not implement authentication and are +unencrypted. You then create a set of configs for the various worker processes. These should be worker configuration files, and should be stored in a dedicated @@ -101,7 +121,8 @@ recommend the use of `systemd` where available: for information on setting up `systemd` to start synapse workers, see [systemd-with-workers](systemd-with-workers). To use `synctl`, see below. -### Using synctl + +#### Using synctl If you want to use `synctl` to manage your synapse processes, you will need to create an an additional configuration file for the master synapse process. That From e36789ab2e200e7724b5581ac5fa6a42cb50937a Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Wed, 6 May 2020 22:28:27 +0100 Subject: [PATCH 2/8] Redis replication configuration --- changelog.d/7446.doc | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelog.d/7446.doc diff --git a/changelog.d/7446.doc b/changelog.d/7446.doc new file mode 100644 index 000000000000..8695022e58bf --- /dev/null +++ b/changelog.d/7446.doc @@ -0,0 +1 @@ +Document Redis replication configuration. From 8610ec13c021275044a0254f3435569638a7116e Mon Sep 17 00:00:00 2001 From: Neil Johnson Date: Wed, 6 May 2020 22:32:55 +0100 Subject: [PATCH 3/8] how to install redis --- docs/workers.md | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index dce82b0e0793..1fe9309f357f 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -121,8 +121,25 @@ recommend the use of `systemd` where available: for information on setting up `systemd` to start synapse workers, see [systemd-with-workers](systemd-with-workers). To use `synctl`, see below. - -#### Using synctl +### Installing Redis +If you are configuring Redis replication, then you need to install Redis +outside of Synapse. + +1. Install redis following the normal procedure for your distro - eg `apt + install redis-server` for debian/ubuntu +2. check it's running and accessible: you should be able to `echo PING | nc -q1 + localhost 6379` and get a response of `+PONG`. +3. Add config to homeserver.yaml: + ```yaml + redis: + enabled: true + ``` + Optional parameters which can go alongside `enabled` are `host`, `port`, + `password`, `dbid`. If you don't know if you need the last two, you + probably don't. +4. That's it. Restart master and all workers. + +### Using synctl If you want to use `synctl` to manage your synapse processes, you will need to create an an additional configuration file for the master synapse process. That From bdc483c76b8299d6fdfdb3b0208debe80a9fa59f Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Thu, 7 May 2020 17:20:13 +0100 Subject: [PATCH 4/8] more redis setup doc --- docs/workers.md | 131 ++++++++++++++++++++++++++---------------------- 1 file changed, 70 insertions(+), 61 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index 1fe9309f357f..7d0373d3e406 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -12,26 +12,21 @@ documenting it here to help admins needing a highly scalable Synapse instance similar to the one running matrix.org. All processes continue to share the same database instance, and as such, -workers only work with postgres based synapse deployments. SQlite should only +workers only work with postgres based synapse deployments. SQLite should only be used for demo purposes and any admin considering workers should already by running postgres. -Originally the workers communicated with the master synapse process via a -synapse-specific TCP protocol called 'replication' - analogous to MySQL or -Postgres style database replication; feeding a stream of relevant data to the +## Master/worker communication + +The workers communicate with the master process via a synapse-specific protocol +called 'replication' - analogous to MySQL- or Postgres-style database +replication - which feeds a stream of relevant data from the master to the workers so they can be kept in sync with the main synapse process and database state. -More recently (v1.13.0) Redis support was added, Redis is an alternative to -direct tcp connections to the master: rather than all the workers connecting to -the master, all the workers and the master connect to redis, which relays -replication commands between processes. This can give a significant cpu saving -on the master and will be a prerequisite for upcoming performance improvements. - -This doc covers details of both approaches (TCP Replication and Redis -Replication), however the TCP Replication implementation will be deprecated in -favour of Redis. - +Additionally, workers may make HTTP requests to the master, to send information +in the other direction. Typically this is used for operations which need to +wait for a reply - such as sending an event. ## Configuration @@ -41,56 +36,44 @@ the correct worker, or to the main synapse instance. Note that this includes requests made to the federation port. See [reverse_proxy.md](reverse_proxy.md) for information on setting up a reverse proxy. -To enable workers, you need to add replication listeners to the master -synapse, in the case of Redis Replication you need only configure an HTTP listener e.g.: - - listeners: - # The HTTP replication port - - port: 9093 - bind_address: '127.0.0.1' - type: http - resources: - - names: [replication] - -For TCP Replication - listeners: - # The TCP replication port - - port: 9092 - bind_address: '127.0.0.1' - type: replication - # The HTTP replication port - - port: 9093 - bind_address: '127.0.0.1' - type: http - resources: - - names: [replication] +To enable workers, you need to add *two* replication listeners to the +main synapse configuration file (`homeserver.yaml`). For example: +```yaml +listeners: + # The TCP replication port + - port: 9092 + bind_address: '127.0.0.1' + type: replication + + # The HTTP replication port + - port: 9093 + bind_address: '127.0.0.1' + type: http + resources: + - names: [replication] +``` Under **no circumstances** should replication API listeners be exposed to the public internet; they currently do not implement authentication and are unencrypted. -You then create a set of configs for the various worker processes. These -should be worker configuration files, and should be stored in a dedicated -subdirectory, to allow synctl to manipulate them. +You shuld then create a set of configs for the various worker processes. Each +worker configuration file inherits the configuration of the main homeserver +configuration file. You can then override configuration specific to that +worker, e.g. the HTTP listener that it provides (if any); logging +configuration; etc. You should minimise the number of overrides though to +maintain a usable config. -Each worker configuration file inherits the configuration of the main homeserver -configuration file. You can then override configuration specific to that worker, -e.g. the HTTP listener that it provides (if any); logging configuration; etc. -You should minimise the number of overrides though to maintain a usable config. - -In the config file for each worker, you must specify the type of worker +In the config file for each worker, you must specify: the type of worker application (`worker_app`). The currently available worker applications are -listed below. You must also specify the replication endpoints that it's talking +listed below. You must also specify the replication endpoints that it talking to on the main synapse process. `worker_replication_host` should specify the host of the main synapse, `worker_replication_port` should point to the TCP replication listener port and `worker_replication_http_port` should point to the HTTP replication port. -Currently, the `event_creator` and `federation_reader` workers require specifying -`worker_replication_http_port`. - -For instance: +For example: worker_app: synapse.app.synchrotron @@ -121,23 +104,49 @@ recommend the use of `systemd` where available: for information on setting up `systemd` to start synapse workers, see [systemd-with-workers](systemd-with-workers). To use `synctl`, see below. -### Installing Redis -If you are configuring Redis replication, then you need to install Redis -outside of Synapse. +### **Experimental** support for replication over redis + +As of Synapse v1.13.0, it is possible to configure synapse to send replication +via a [Redis pub/sub channel](https://redis.io/topics/pubsub). This is an +alternative to direct tcp connections to the master: rather than all the +workers connecting to the master, all the workers and the master connect to +Redis, which relays replication commands between processes. This can give a +significant cpu saving on the master and will be a prerequisite for upcoming +performance improvements. + +Note that this support is currently experimental; you may experience lost +messages and similar problems! It is strongly recommended that admins setting +up workers for the first time use direct TCP replication as above. + +To configure Synapse to use Redis: -1. Install redis following the normal procedure for your distro - eg `apt - install redis-server` for debian/ubuntu -2. check it's running and accessible: you should be able to `echo PING | nc -q1 +1. Install Redis following the normal procedure for your distribution - for + example, on Debian, `apt install redis-server`. (It is safe to use an + existing Redis deployment if you have one: we use a pub/sub stream named + according to the `server_name` of your synapse server.) +2. Check Redis is running and accessible: you should be able to `echo PING | nc -q1 localhost 6379` and get a response of `+PONG`. -3. Add config to homeserver.yaml: +3. Install the python prerequisites. If you installed synapse into a + virtualenv, this can be done with: + ```sh + pip install matrix-synapse[redis] + ``` + The debian packages from matrix.org already include the required + dependencies. +4. Add config to the shared configuration (`homeserver.yaml`): ```yaml redis: enabled: true ``` Optional parameters which can go alongside `enabled` are `host`, `port`, - `password`, `dbid`. If you don't know if you need the last two, you - probably don't. -4. That's it. Restart master and all workers. + `password`. Normally none of these are required. +5. Restart master and all workers. + +Once redis replication is in use, `worker_replication_port` is redundant and +can be removed from the worker configuration files. Similarly, the +configuration for the `listener` for the TCP replication port can be removed +from the main configuration file. Note that the *HTTP* replication port is +still required. ### Using synctl From 2f0947acbb108448f09412034e0d2db3cc40f00e Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Thu, 7 May 2020 17:21:38 +0100 Subject: [PATCH 5/8] changelog --- changelog.d/7446.doc | 1 - changelog.d/7446.feature | 1 + 2 files changed, 1 insertion(+), 1 deletion(-) delete mode 100644 changelog.d/7446.doc create mode 100644 changelog.d/7446.feature diff --git a/changelog.d/7446.doc b/changelog.d/7446.doc deleted file mode 100644 index 8695022e58bf..000000000000 --- a/changelog.d/7446.doc +++ /dev/null @@ -1 +0,0 @@ -Document Redis replication configuration. diff --git a/changelog.d/7446.feature b/changelog.d/7446.feature new file mode 100644 index 000000000000..ce6140fdd111 --- /dev/null +++ b/changelog.d/7446.feature @@ -0,0 +1 @@ +Add support for running replication over Redis when using workers. From 426c8ec404ced1e571c29d9b9ee2304f79582560 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Thu, 7 May 2020 17:25:52 +0100 Subject: [PATCH 6/8] minor edits --- docs/workers.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index 7d0373d3e406..021ae4dea18a 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -9,12 +9,12 @@ horizontally independently. Synapse's worker support is under active development and subject to change as we attempt to rapidly scale ever larger Synapse instances. However we are documenting it here to help admins needing a highly scalable Synapse instance -similar to the one running matrix.org. +similar to the one running `matrix.org`. All processes continue to share the same database instance, and as such, -workers only work with postgres based synapse deployments. SQLite should only +workers only work with PostgreSQL based synapse deployments. SQLite should only be used for demo purposes and any admin considering workers should already by -running postgres. +running PostgreSQL. ## Master/worker communication @@ -54,22 +54,21 @@ listeners: - names: [replication] ``` -Under **no circumstances** should replication API listeners be exposed to -the public internet; they currently do not implement authentication and are -unencrypted. +Under **no circumstances** should these replication API listeners be exposed to +the public internet; they have no authentication and are unencrypted. -You shuld then create a set of configs for the various worker processes. Each +You should then create a set of configs for the various worker processes. Each worker configuration file inherits the configuration of the main homeserver configuration file. You can then override configuration specific to that worker, e.g. the HTTP listener that it provides (if any); logging configuration; etc. You should minimise the number of overrides though to maintain a usable config. -In the config file for each worker, you must specify: the type of worker +In the config file for each worker, you must specify the type of worker application (`worker_app`). The currently available worker applications are -listed below. You must also specify the replication endpoints that it talking -to on the main synapse process. `worker_replication_host` should specify the -host of the main synapse, `worker_replication_port` should point to the TCP +listed below. You must also specify the replication endpoints that it should +talk to on the main synapse process. `worker_replication_host` should specify +the host of the main synapse, `worker_replication_port` should point to the TCP replication listener port and `worker_replication_http_port` should point to the HTTP replication port. From 7e2149aab23d35772401318c1517d4864c33187f Mon Sep 17 00:00:00 2001 From: Richard van der Hoff Date: Thu, 7 May 2020 17:30:55 +0100 Subject: [PATCH 7/8] more minor edits --- docs/workers.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index 021ae4dea18a..3aa1b67977d1 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -12,17 +12,16 @@ documenting it here to help admins needing a highly scalable Synapse instance similar to the one running `matrix.org`. All processes continue to share the same database instance, and as such, -workers only work with PostgreSQL based synapse deployments. SQLite should only -be used for demo purposes and any admin considering workers should already by +workers only work with PostgreSQL-based Synapse deployments. SQLite should only +be used for demo purposes and any admin considering workers should already be running PostgreSQL. ## Master/worker communication -The workers communicate with the master process via a synapse-specific protocol -called 'replication' - analogous to MySQL- or Postgres-style database -replication - which feeds a stream of relevant data from the master to the -workers so they can be kept in sync with the main synapse process and database -state. +The workers communicate with the master process via a Synapse-specific protocol +called 'replication' (analogous to MySQL- or Postgres-style database +replication) which feeds a stream of relevant data from the master to the +workers so they can be kept in sync with the master process and database state. Additionally, workers may make HTTP requests to the master, to send information in the other direction. Typically this is used for operations which need to @@ -37,7 +36,7 @@ requests made to the federation port. See [reverse_proxy.md](reverse_proxy.md) for information on setting up a reverse proxy. To enable workers, you need to add *two* replication listeners to the -main synapse configuration file (`homeserver.yaml`). For example: +main Synapse configuration file (`homeserver.yaml`). For example: ```yaml listeners: @@ -105,9 +104,9 @@ recommend the use of `systemd` where available: for information on setting up ### **Experimental** support for replication over redis -As of Synapse v1.13.0, it is possible to configure synapse to send replication +As of Synapse v1.13.0, it is possible to configure Synapse to send replication via a [Redis pub/sub channel](https://redis.io/topics/pubsub). This is an -alternative to direct tcp connections to the master: rather than all the +alternative to direct TCP connections to the master: rather than all the workers connecting to the master, all the workers and the master connect to Redis, which relays replication commands between processes. This can give a significant cpu saving on the master and will be a prerequisite for upcoming @@ -144,7 +143,7 @@ To configure Synapse to use Redis: Once redis replication is in use, `worker_replication_port` is redundant and can be removed from the worker configuration files. Similarly, the configuration for the `listener` for the TCP replication port can be removed -from the main configuration file. Note that the *HTTP* replication port is +from the main configuration file. Note that the HTTP replication port is still required. ### Using synctl From 2d98123bf40295ca05425f546aeae20714c3c394 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Mon, 11 May 2020 13:19:54 +0100 Subject: [PATCH 8/8] syntax highlighting --- docs/workers.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/docs/workers.md b/docs/workers.md index 3aa1b67977d1..7512eff43a77 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -73,21 +73,23 @@ the HTTP replication port. For example: - worker_app: synapse.app.synchrotron +```yaml +worker_app: synapse.app.synchrotron - # The replication listener on the synapse to talk to. - worker_replication_host: 127.0.0.1 - worker_replication_port: 9092 - worker_replication_http_port: 9093 +# The replication listener on the synapse to talk to. +worker_replication_host: 127.0.0.1 +worker_replication_port: 9092 +worker_replication_http_port: 9093 - worker_listeners: - - type: http - port: 8083 - resources: - - names: - - client +worker_listeners: + - type: http + port: 8083 + resources: + - names: + - client - worker_log_config: /home/matrix/synapse/config/synchrotron_log_config.yaml +worker_log_config: /home/matrix/synapse/config/synchrotron_log_config.yaml +``` ...is a full configuration for a synchrotron worker instance, which will expose a plain HTTP `/sync` endpoint on port 8083 separately from the `/sync` endpoint provided