Additional functionality for declaring worker types in Complement

[realtyem]

TL;DR: The Complement Docker image includes the capability of defining what workers 'types' should be tested against in an environmental variable. All workers are now generic workers. Therefore, multiple types of workers can be merged onto a single worker. Enable this for the scripting that starts up Complement(and the Dockerfile-workers) image. Also add in a worker multiplier function, to allow having more than one worker type without repeating it in the environmental variable.

The Long Version:
This PR adds 3 new capabilities to the Synapse-workers Docker image.

1. Multiply the number of workers declared by appending :# (where # is a positive integer) to the end of the worker type definition. For example:

   • SYNAPSE_WORKER_TYPES='event_persister:2'

   would be the functional equivalent to

   • SYNAPSE_WORKER_TYPES='event_persister, event_persister'

   It will not allow for duplicating worker types that can not have more than one, like background_worker or any of the stream writers(excepting event_persister).

2. Give each worker type definition a custom name by prepending name= to the front of the individual worker type. This will propagate to anywhere the worker's name is used, as it is written into each given worker's yaml configuration. This holds true for the docker logs and nginx configuration. For example:

   • SYNAPSE_WORKER_TYPES='federation_sender, federation_inbound'

   can now be

   • SYNAPSE_WORKER_TYPES='thelma=federation_sender, louise=federation_inbound'

   Custom names with spaces is not allowed, as it will lead to FileNotFoundError's. The error message that is logged suggests using an underscore instead.

3. Allow merging separate worker types into a single worker using a + between the worker types to be combined. Since the functionality for the majority of the worker types is based on URL endpoint routed to the worker, this is a simple merging of those endpoints before being added into the reverse proxy configuration. Other worker types use special parameters in the shared yaml configuration(such as stream_writers, instance_map, update_user_directory_from_worker, etc), these were taken into account as well. The first worker to include any of the *_from_worker option gets nominated for the job. Worker types that include special 'Maps' or 'Lists' will be correctly maintained. Examples:

   • SYNAPSE_WORKER_TYPES='user_dir+federation_reader'
   • SYNAPSE_WORKER_TYPES='media_repository+pusher+event_creator'
   • SYNAPSE_WORKER_TYPES='client_reader+federation_sender, federation_inbound+pusher'

   and my favorite, which makes use of capabilities 1 and 2 above:

   • SYNAPSE_WORKER_TYPES='stream_writers=account_data+presence+receipts+to_device+typing, event_persister:2'

   This effectively adds all the 5 stream writer workers to a Complement test run, but is only using the resources of 1 additional worker and is not making a ridiculously long name in the logs.

To pass tests, Needs:

• Experimental MSC3890 Implementation: Fix deleting account data when using an account data writer worker #14869

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file.
• Pull request includes a sign off
• Code style is correct
  (run the linters)

Signed-off-by: Jason Little realtyem@gmail.com

[reivilibre]

sorry for taking the time on this one.

My main comment is just that this is adding to an already quite long-winded chunk of code; some of that could be easily improved though.

[reivilibre]

I know it was already fairly poor before you came to it, but generate_worker_files is now monstruously long.

It might be better if the logic for parsing and validation for the worker type string was pulled out into a function that accepts the worker description string and returns

Dict[str, Set[str]] — mapping from worker name to set of worker roles

?

I'd try and keep it quite concise/punchy because it's hard to read such long chains of processing.

[realtyem]

Hopefully it's closer to what you asked for. I did my best.