Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Additional functionality for declaring worker types in Complement #14921

Merged
merged 33 commits into from
Mar 14, 2023

Commits on Jan 23, 2023

  1. Allow for a multipler to be added to a requested worker.

    This allows for appending a ':' and a positive integer to a requested worker type for the SYNAPSE_WORKER_TYPES
    environment variable(and likewise, the WORKER_TYPES) used in Complement. For example:
    
    'event_persister:2' would be yield identical results of 'event_persister, event_persister'
    realtyem committed Jan 23, 2023
    Configuration menu
    Copy the full SHA
    4647d59 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2023

  1. Allow for worker types to be merged and given a name.

    In the Synapse-worker docker image, multiple worker types are defined but only allowed to be that one single definition.
    This makes it so that any two(or more) worker types can be merged into a single worker. For example:
    1. If you wish to have one worker with both client_reader and federation_sender functions enabled, you can set
    SYNAPSE_WORKER_TYPES='client_reader+federation_sender'
    2. If you wish to have all stream_writers(excepting event_persister) you can set
    SYNAPSE_WORKER_TYPES='account_data+presence+receipts+to_device+typing'
    
    Multiple types can be combined, but some error checking to dis-allow multiples of worker types that shouldn't be
    enabled more than once has been added. For example:
    SYNAPSE_WORKER_TYPES='background_worker+event_persister, background_worker+event_persister' will not work as
    background_worker is only allowed to have a single worker for the entire deployment.
    
    Giving a worker or a combination of workers a custom name is as simple as adding the name then an equal sign in front
    of the given worker type(s). For example:
    SYNAPSE_WORKER_TYPES='alice=federation_reader'
    or
    SYNAPSE_WORKER_TYPES='bob=federation_inbound+federation_sender'
    or
    SYNAPSE_WORKER_TYPES='charlie=event_persister:2, derek=media_repository + pusher + user_dir + appservice + event_creator'
    realtyem committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    e29ef6f View commit details
    Browse the repository at this point in the history
  2. Install examples in start_for_complement.sh.

    Instead of adding stream writer workers individually, utilize new functionality to make one worker with all included. Also utilitize new shorthand for event_persister's.
    realtyem committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    8538cb6 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    02178ce View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2023

  1. Caught a typo

    realtyem committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    bd6b73e View commit details
    Browse the repository at this point in the history
  2. Changelog

    realtyem committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    c858d45 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e00989f View commit details
    Browse the repository at this point in the history

Commits on Feb 5, 2023

  1. Configuration menu
    Copy the full SHA
    497f01d View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2023

  1. Configuration menu
    Copy the full SHA
    beb2531 View commit details
    Browse the repository at this point in the history

Commits on Feb 19, 2023

  1. Apply changes from review:

    1. Turn 'placeholder_name' into a magic constant.
    2. Fix casting on a worker_type_list from 'list' to 'List[str]'.
    3. In 'insert_worker_name_for_shared_config', use new WORKER_PLACEHOLDER_NAME constant and reduce complexity by using value from items() directly.
    4. In 'is_sharding_allowed_for_worker_type', simplify the return value.
    5. Adjust 'split_string_and_strip' to allow for using the 'maxsplit' kwarg on 'strip' and update comment.
    6. Remove function 'increment_counter' and use 'defaultdict' instead to reduce complexity.
    7. Remove function 'is_name_allowed_for_worker', as this can be accomplished with an 'if' statement and only checked for a worker base name in a
        dict. Adjust 'generate_worker_files' to accommadate.
    8. Rename a few dict's to remove misnomer's and adjust comments for extra clarity.
    9. In 'generate_worker_files':
      a. Use 'ValueError' exception handling to catch additional errors when parsing an integer and simplify counting of appended worker_type's.
      b. Add additional error cases to check(single and double quotes, spaces, and last character being an integer) for potential file name/nginx issues.
      c. Adjust various variable names to improve readability.
    10. Update various comments throughout.
    realtyem committed Feb 19, 2023
    Configuration menu
    Copy the full SHA
    8d1b37a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f964a1a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    aba5585 View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2023

  1. Configuration menu
    Copy the full SHA
    3a7df06 View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2023

  1. Apply changes from review.

    1. Factor processing of all worker types from the environment out and away from
        generating config files.
    2. Fix up early templating of 'worker_config' to remove boilerplate.
    3. Wrangle nginx upstream processing to accommodate odd combinations for overlapping
        workers. e.g. 'user_dir, user_dir+presence' should be able to handle the(in this case
        single) endpoint over either worker. I think this was promised in a previous commit,
        consider it delivered.
    4. Update a bunch of comments, and adjust some pre-existing to fit inside the line
        length count of 88(aka, make the green squiggles go away)
    5. Move processing of unusable characters for worker names to a later position so it
        will check the name produced if one was not requested as well.
    realtyem committed Mar 3, 2023
    Configuration menu
    Copy the full SHA
    47a965f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5dc3318 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2023

  1. Configuration menu
    Copy the full SHA
    dd9773c View commit details
    Browse the repository at this point in the history
  2. Disambiguate a confusing mess of 'worker_roles', 'workers_roles', 'wo…

    …rker_type', and similar and that should have been 'worker_types' when in use by sets and a couple other places.
    realtyem committed Mar 5, 2023
    Configuration menu
    Copy the full SHA
    4442c0a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    98da503 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2023

  1. Update comments and a few other texty bits:

    1. Update comments and edit(or create) docstrings and a single type hint that was incomplete(split_and_strip_string()).
    2. Rename parse_worker_types_from_env to parse_worker_types as it's not coming from the environment.
    realtyem committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    b8741d7 View commit details
    Browse the repository at this point in the history
  2. Update how worker_type specific options are merged into shared_config…

    …, and update comments to match.
    realtyem committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    204f61e View commit details
    Browse the repository at this point in the history
  3. Remove condition based on number of ports

    ...so that even with a single port, a worker gets it's own upstream. Nginx says this can
    lead to performance improvements if some features are enabled. Revisit this later.
    realtyem committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    7c32bdb View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    0cf5e80 View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2023

  1. Try and apply review comments about sanitizing names.

    1. Update a few comments for clarity.
    2. Apply a replace() to remove whitespace in a requested worker name and use an underscore instead.
    3. Stop-error if requesting a worker name, but forgetting to actually put one in.
    4. If a worker name isn't requested, just use the set of worker types, sorted and concatenated with a hyphen. This avoids having to sanitize the worker name in this case.
    5. Add regex to avoid nasty name surprises.
    realtyem committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    e2fecab View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    97a0af0 View commit details
    Browse the repository at this point in the history
  3. Fix log message and the condition it fires on to reflect it only happ…

    …ens if more than one worker type for a given worker is requested.
    realtyem committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    9f3f9ab View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    65de783 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    aeefc5d View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    b5a66b4 View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2023

  1. Configuration menu
    Copy the full SHA
    54c23c9 View commit details
    Browse the repository at this point in the history
  2. Remove unnecessary comment.

    realtyem committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    32c04bb View commit details
    Browse the repository at this point in the history
  3. Tweak regex to not allow final character to be a number, and adjust w…

    …hitespace on error message.
    realtyem committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    78a43c8 View commit details
    Browse the repository at this point in the history
  4. Update comment

    realtyem committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    8ff80d7 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    6b90ccb View commit details
    Browse the repository at this point in the history