Merge tag 'v1.46.0' into merge_synapse_v1_46_0

Synapse 1.46.0 (2021-11-02)
===========================

The cause of the [performance regression affecting Synapse 1.44](matrix-org#11049) has been identified and fixed. ([\matrix-org#11177](matrix-org#11177))

Bugfixes
--------

- Fix a bug introduced in v1.46.0rc1 where URL previews of some XML documents would fail. ([\matrix-org#11196](matrix-org#11196))

Synapse 1.46.0rc1 (2021-10-27)
==============================

Features
--------

- Add support for Ubuntu 21.10 "Impish Indri". ([\matrix-org#11024](matrix-org#11024))
- Port the Password Auth Providers module interface to the new generic interface. ([\matrix-org#10548](matrix-org#10548), [\matrix-org#11180](matrix-org#11180))
- Experimental support for the thread relation defined in [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\matrix-org#11088](matrix-org#11088), [\matrix-org#11181](matrix-org#11181), [\matrix-org#11192](matrix-org#11192))
- Users admin API can now also modify user type in addition to allowing it to be set on user creation. ([\matrix-org#11174](matrix-org#11174))

Bugfixes
--------

- Newly-created public rooms are now only assigned an alias if the room's creation has not been blocked by permission settings. Contributed by @AndrewFerr. ([\matrix-org#10930](matrix-org#10930))
- Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state. ([\matrix-org#11001](matrix-org#11001), [\matrix-org#11009](matrix-org#11009), [\matrix-org#11012](matrix-org#11012))
- Fix 500 error on `/messages` when the server accumulates more than 5 backwards extremities at a given depth for a room. ([\matrix-org#11027](matrix-org#11027))
- Fix a bug where setting a user's `external_id` via the admin API returns 500 and deletes user's existing external mappings if that external ID is already mapped. ([\matrix-org#11051](matrix-org#11051))
- Fix a long-standing bug where users excluded from the user directory were added into the directory if they belonged to a room which became public or private. ([\matrix-org#11075](matrix-org#11075))
- Fix a long-standing bug when attempting to preview URLs which are in the `windows-1252` character encoding. ([\matrix-org#11077](matrix-org#11077), [\matrix-org#11089](matrix-org#11089))
- Fix broken export-data admin command and add test script checking the command to CI. ([\matrix-org#11078](matrix-org#11078))
- Show an error when timestamp in seconds is provided to the `/purge_media_cache` Admin API. ([\matrix-org#11101](matrix-org#11101))
- Fix local users who left all their rooms being removed from the user directory, even if the `search_all_users` config option was enabled. ([\matrix-org#11103](matrix-org#11103))
- Fix a bug which caused the module API's `get_user_ip_and_agents` function to always fail on workers. `get_user_ip_and_agents` was introduced in 1.44.0 and did not function correctly on worker processes at the time. ([\matrix-org#11112](matrix-org#11112))
- Identity server connection is no longer ignoring `ip_range_whitelist`. ([\matrix-org#11120](matrix-org#11120))
- Fix a bug introduced in Synapse 1.45.0 breaking the configuration file parsing script. ([\matrix-org#11145](matrix-org#11145))
- Fix a performance regression introduced in 1.44.0 which could cause client requests to time out when making large numbers of outbound requests. ([\matrix-org#11177](matrix-org#11177), [\matrix-org#11190](matrix-org#11190))
- Resolve and share `state_groups` for all [MSC2716](matrix-org/matrix-spec-proposals#2716) historical events in batch. ([\matrix-org#10975](matrix-org#10975))

Improved Documentation
----------------------

- Fix broken links relating to module API deprecation in the upgrade notes. ([\matrix-org#11069](matrix-org#11069))
- Add more information about what happens when a user is deactivated. ([\matrix-org#11083](matrix-org#11083))
- Clarify the the sample log config can be copied from the documentation without issue. ([\matrix-org#11092](matrix-org#11092))
- Update the admin API documentation with an updated list of the characters allowed in registration tokens. ([\matrix-org#11093](matrix-org#11093))
- Document Synapse's behaviour when dealing with multiple modules registering the same callbacks and/or handlers for the same HTTP endpoints. ([\matrix-org#11096](matrix-org#11096))
- Fix instances of `[example]{.title-ref}` in the upgrade documentation as a result of prior RST to Markdown conversion. ([\matrix-org#11118](matrix-org#11118))
- Document the version of Synapse each module callback was introduced in. ([\matrix-org#11132](matrix-org#11132))
- Document the version of Synapse that introduced each module API method. ([\matrix-org#11183](matrix-org#11183))

Internal Changes
----------------
- Fix spurious warnings about losing the logging context on the `ReplicationCommandHandler` when losing the replication connection. ([\matrix-org#10984](matrix-org#10984))
- Include rejected status when we log events. ([\matrix-org#11008](matrix-org#11008))
- Add some extra logging to the event persistence code. ([\matrix-org#11014](matrix-org#11014))
- Rearrange the internal workings of the incremental user directory updates. ([\matrix-org#11035](matrix-org#11035))
- Fix a long-standing bug where users excluded from the directory could still be added to the `users_who_share_private_rooms` table after a regular user joins a private room. ([\matrix-org#11143](matrix-org#11143))
- Add and improve type hints. ([\matrix-org#10972](matrix-org#10972), [\matrix-org#11055](matrix-org#11055), [\matrix-org#11066](matrix-org#11066), [\matrix-org#11076](matrix-org#11076), [\matrix-org#11095](matrix-org#11095), [\matrix-org#11109](matrix-org#11109), [\matrix-org#11121](matrix-org#11121), [\matrix-org#11146](matrix-org#11146))
- Mark the Synapse package as containing type annotations and fix export declarations so that Synapse pluggable modules may be type checked against Synapse. ([\matrix-org#11054](matrix-org#11054))
- Remove dead code from `MediaFilePaths`. ([\matrix-org#11056](matrix-org#11056))
- Be more lenient when parsing oEmbed response versions. ([\matrix-org#11065](matrix-org#11065))
- Create a separate module for the retention configuration. ([\matrix-org#11070](matrix-org#11070))
- Clean up some of the federation event authentication code for clarity. ([\matrix-org#11115](matrix-org#11115), [\matrix-org#11116](matrix-org#11116), [\matrix-org#11122](matrix-org#11122))
- Add docstrings and comments to the application service ephemeral event sending code. ([\matrix-org#11138](matrix-org#11138))
- Update the `sign_json` script to support inline configuration of the signing key. ([\matrix-org#11139](matrix-org#11139))
- Fix broken link in the docker image README. ([\matrix-org#11144](matrix-org#11144))
- Always dump logs from unit tests during CI runs. ([\matrix-org#11068](matrix-org#11068))
- Add tests for `MediaFilePaths` class. ([\matrix-org#11057](matrix-org#11057))
- Simplify the user admin API tests. ([\matrix-org#11048](matrix-org#11048))
- Add a test for the workaround introduced in [\matrix-org#11042](matrix-org#11042) concerning the behaviour of third-party rule modules and `SynapseError`s. ([\matrix-org#11071](matrix-org#11071))

.ci/patch_for_twisted_trunk.sh

@@ -0,0 +1,8 @@
+#!/bin/sh
+
+# replaces the dependency on Twisted in `python_dependencies` with trunk.
+
+set -e
+cd "$(dirname "$0")"/..
+
+sed -i -e 's#"Twisted.*"#"Twisted @ git+https://github.com/twisted/twisted"#' synapse/python_dependencies.py

.buildkite/postgres-config.yaml → .ci/postgres-config.yaml

@@ -3,19 +3,17 @@
 # CI's Docker setup at the point where this file is considered.
 server_name: "localhost:8800"
 
-signing_key_path: "/src/.buildkite/test.signing.key"
+signing_key_path: ".ci/test.signing.key"
 
 report_stats: false
 
 database:
   name: "psycopg2"
   args:
     user: postgres
-    host: postgres
+    host: localhost
     password: postgres
     database: synapse
 
 # Suppress the key server warning.
-trusted_key_servers:
-  - server_name: "matrix.org"
-suppress_key_server_warning: true
+trusted_key_servers: []

synapse/replication/slave/storage/transactions.py → .ci/scripts/postgres_exec.py

@@ -1,5 +1,5 @@
-# -*- coding: utf-8 -*-
-# Copyright 2015, 2016 OpenMarket Ltd
+#!/usr/bin/env python
+# Copyright 2019 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,10 +13,19 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.storage.databases.main.transactions import TransactionStore
+import sys
 
-from ._base import BaseSlavedStore
+import psycopg2
 
+# a very simple replacment for `psql`, to make up for the lack of the postgres client
+# libraries in the synapse docker image.
 
-class SlavedTransactionStore(TransactionStore, BaseSlavedStore):
-    pass
+# We use "postgres" as a database because it's bound to exist and the "synapse" one
+# doesn't exist yet.
+db_conn = psycopg2.connect(
+    user="postgres", host="localhost", password="postgres", dbname="postgres"
+)
+db_conn.autocommit = True
+cur = db_conn.cursor()
+for c in sys.argv[1:]:
+    cur.execute(c)

.ci/scripts/test_export_data_command.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+
+# Test for the export-data admin command against sqlite and postgres
+
+set -xe
+cd `dirname $0`/../..
+
+echo "--- Install dependencies"
+
+# Install dependencies for this test.
+pip install psycopg2
+
+# Install Synapse itself. This won't update any libraries.
+pip install -e .
+
+echo "--- Generate the signing key"
+
+# Generate the server's signing key.
+python -m synapse.app.homeserver --generate-keys -c .ci/sqlite-config.yaml
+
+echo "--- Prepare test database"
+
+# Make sure the SQLite3 database is using the latest schema and has no pending background update.
+scripts/update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates
+
+# Run the export-data command on the sqlite test database
+python -m synapse.app.admin_cmd -c .ci/sqlite-config.yaml  export-data @anon-20191002_181700-832:localhost:8800 \
+--output-directory /tmp/export_data
+
+# Test that the output directory exists and contains the rooms directory
+dir="/tmp/export_data/rooms"
+if [ -d "$dir" ]; then
+  echo "Command successful, this test passes"
+else
+  echo "No output directories found, the command fails against a sqlite database."
+  exit 1
+fi
+
+# Create the PostgreSQL database.
+.ci/scripts/postgres_exec.py "CREATE DATABASE synapse"
+
+# Port the SQLite databse to postgres so we can check command works against postgres
+echo "+++ Port SQLite3 databse to postgres"
+scripts/synapse_port_db --sqlite-database .ci/test_db.db --postgres-config .ci/postgres-config.yaml
+
+# Run the export-data command on postgres database
+python -m synapse.app.admin_cmd -c .ci/postgres-config.yaml  export-data @anon-20191002_181700-832:localhost:8800 \
+--output-directory /tmp/export_data2
+
+# Test that the output directory exists and contains the rooms directory
+dir2="/tmp/export_data2/rooms"
+if [ -d "$dir2" ]; then
+  echo "Command successful, this test passes"
+else
+  echo "No output directories found, the command fails against a postgres database."
+  exit 1
+fi

.buildkite/scripts/test_old_deps.sh → .ci/scripts/test_old_deps.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 
-# this script is run by buildkite in a plain `bionic` container; it installs the
+# this script is run by GitHub Actions in a plain `bionic` container; it installs the
 # minimal requirements for tox and hands over to the py3-old tox environment.
 
 set -ex

.ci/scripts/test_synapse_port_db.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+#
+# Test script for 'synapse_port_db'.
+#   - sets up synapse and deps
+#   - runs the port script on a prepopulated test sqlite db
+#   - also runs it against an new sqlite db
+
+
+set -xe
+cd `dirname $0`/../..
+
+echo "--- Install dependencies"
+
+# Install dependencies for this test.
+pip install psycopg2 coverage coverage-enable-subprocess
+
+# Install Synapse itself. This won't update any libraries.
+pip install -e .
+
+echo "--- Generate the signing key"
+
+# Generate the server's signing key.
+python -m synapse.app.homeserver --generate-keys -c .ci/sqlite-config.yaml
+
+echo "--- Prepare test database"
+
+# Make sure the SQLite3 database is using the latest schema and has no pending background update.
+scripts/update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates
+
+# Create the PostgreSQL database.
+.ci/scripts/postgres_exec.py "CREATE DATABASE synapse"
+
+echo "+++ Run synapse_port_db against test database"
+coverage run scripts/synapse_port_db --sqlite-database .ci/test_db.db --postgres-config .ci/postgres-config.yaml
+
+# We should be able to run twice against the same database.
+echo "+++ Run synapse_port_db a second time"
+coverage run scripts/synapse_port_db --sqlite-database .ci/test_db.db --postgres-config .ci/postgres-config.yaml
+
+#####
+
+# Now do the same again, on an empty database.
+
+echo "--- Prepare empty SQLite database"
+
+# we do this by deleting the sqlite db, and then doing the same again.
+rm .ci/test_db.db
+
+scripts/update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates
+
+# re-create the PostgreSQL database.
+.ci/scripts/postgres_exec.py \
+  "DROP DATABASE synapse" \
+  "CREATE DATABASE synapse"
+
+echo "+++ Run synapse_port_db against empty database"
+coverage run scripts/synapse_port_db --sqlite-database .ci/test_db.db --postgres-config .ci/postgres-config.yaml

.buildkite/sqlite-config.yaml → .ci/sqlite-config.yaml

@@ -3,16 +3,14 @@
 # schema and run background updates on it.
 server_name: "localhost:8800"
 
-signing_key_path: "/src/.buildkite/test.signing.key"
+signing_key_path: ".ci/test.signing.key"
 
 report_stats: false
 
 database:
   name: "sqlite3"
   args:
-    database: ".buildkite/test_db.db"
+    database: ".ci/test_db.db"
 
 # Suppress the key server warning.
-trusted_key_servers:
-  - server_name: "matrix.org"
-suppress_key_server_warning: true
+trusted_key_servers: []

.ci/twisted_trunk_build_failed_issue_template.md

@@ -0,0 +1,4 @@
+---
+title: CI run against Twisted trunk is failing
+---
+See https://github.com/{{env.GITHUB_REPOSITORY}}/actions/runs/{{env.GITHUB_RUN_ID}}

.ci/worker-blacklist

@@ -0,0 +1,2 @@
+# This file serves as a blacklist for SyTest tests that we expect will fail in
+# Synapse when run under worker mode. For more details, see sytest-blacklist.