merge dev into mls #3455
Merged
merge dev into mls #3455
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Make note about poetry clearer.
* Migrated notifications. * Undo EmptyResponse changes * Revert unnecessary test changes. * Revert mls welcome. * Added e2e test removing temp client across backends * Add integration test for client deletion * Removed redundant e2e * Updated nix for integration * Migrate on-msg-sent to queue * delete/adjust tests. * backend-notification-pusher: Dynamically discover remote domains (#3366) * Bumped timeout * Persist typing indicator * Reverted typing indicator back to sync RPC * Deleted outdated tests * [WPB-2789] Use EmptyResponse consistently across Galley federation endpoints (#3365) * Use EmptyResponse consistently for galley fed calls * Amended doc for empty response. * Fixed some failing tests. * Temporary remove test. * Restore EmptyResponse instead of (). * Restore tests removed by accident --------- Co-authored-by: Akshay Mankar <akshay@wire.com>
Stern: Reduce batch size to 50 when GetNotifications
…other backend (#3378) * WIP * FS-1115: Fixes for some tests * Merge config file in with db in `GET /federation/remotes`. * FS-1115: Fixing build errors, updating tests, formatting * Don't allow removing remote domains from config file via rest api. * FS-1115: Fixing compile errors on a new test * FS-1115: Fixing the federation CRUD test * FS-1115: Adding federation domain updates to brig and cannon * Docs (draft), release notes. * (Slightly) better api error on fail-to-delete. * Breadcrumbs. * Better breadcrumbs. We don't know which release this will land in yet, PR is more future-proof. * Polish docs. * afterthought. * Update services/brig/test/integration/API/Federation.hs * More tests. * Remove a lying comment. * Refactor: keep the semantics of `_runSettings` intact. (This is part of the `Opts` data structure read from a file. it's very confusing to start updating part of that on the fly, and ignoring whatever comes from the config file.) * Remove a lying comment. * Fixup / WIP * Fixup * I take it back! * more docs. * rm bogus TODO, add a non-bogus one :) * revert changes of status code in integration tests. (Double-check if we can't keep the prod code behavior intact to avoid upgrade / fed-to-fed api / client api issues.) * Various updates, mainly moving some common code into wire-api and minimising types that are used in several common paths. * FS-1115: More common code for federation domain updates * Use tinylog instead of `print`. * sanitize-pr * Use retry instead of threadDelay-loop. * FS-1115: Updating design docs on how update intervals are supplied * Clarify source comment. * FS-1115: Cleaning up un-needed changes to options * update docs.wire.com * docs. * docs. * Fix: `/i/user/meta-info` (stern) (#3281) * Handle race conditions in /integration (#3278) * Update docs.wire.com (#3284) * Deprecate start-services-only.sh * Fix outdated docs on how to run swagger docs locally. * Restore deleted scripts (#3287) * Refactor federation domain configuration. - move strategy config from federator to brig. - remove domain list from strategy type. - add 'none' strategy (more explicit that 'list' with empty list, but both still work) * Fixup * Complete CRUD api for federator remotes (Update is missing) [WIP]. * Complete CRUD api for federator remotes (Update is missing): tests. * docs. * Cleanup * FS-1179: Setting up a location in galley for processing deleted domains NOTE: Broken! Currently not compiling Adding a new query to cassandra for getting remote members and conversation IDs that they are in, setting up loops for removing these members and potentially conversations using existing APIs. * FS-1179: Fixing the polysemy error. Picking an error type and documenting what might need to be done for better error mapping. * FS-1179: Initial processing for deleting remote domains. Setting up Galley to remove remote users from local conversations, local users from remote conversations, and to delete all user connections for the federation domain that is being dropped. * FS-1179: Fixing an error http reponse code. * wip * Remove dead code. * nit-pick. * Typo. * Fine-tune logging. * Fix haddocks. * Implement put. * Clariy updateFrequency everywhere. * nit-pick. * Fixup * Cleanup * Implement put. (For real this time.) * Move integration tests to /integration. * FS-1179: Work in progress. Tests are broken due to timing issues * FS-1179: Fixing an errant delete * FS-1179: Commiting and switching to FS-1115 * ... * wip * ... * FS-1115: Setting up calls to Brig from integration tests. JSON, calling, etc. * ... * ... * ... * ... * ... * ... * Fix * ... * WIP: Chasing down cache issues between some bug fixes * ... * ... * ... * ... * Tests are passing! * process leif's feedback. * ... * Fixing compile issues after a merge * Updating templates * Removing a JSON roundtrip, better using aeson. * Moving more of the federation domain update code into wire-api. Tests are broken with these changes. * wip * sanitize-pr * FS-1115: Updating brig integration config to help tests * s/AllowList/AllowDynamic/g * rm trailing whitespace. * docs * Updating with PR feedback * Have federation domain tests use MakesValue more * FS-1115: Removing more FedConn type specific code from federation tests * More code leaning on typeclasses * Pre-emptive rework before merging * Mark flaky test case. * docs. * Formatting * FS-1179: Setting up the receiving RabbitMQ loop. Setup a reading loop for RabbitMQ to get the domains that need to be deleted. We're using a queue here so that we have a non-volatile store for these domains that can automatically handle galley instances dying or being scaled down. * wip * FS-1179: Refactoring the RabbitMQ code Refactoring RabbitMQ code so that there is a single channel for both production and consumption of messages. There is also a function for "simple" processing where a message queuing system isn't available, but it still needs to have persistence and task co-ordination defined for it. This may not be a problem if we decide to mandate the use of RabbitMQ, or a similar queue. * Code formatting * FS-1179: Fixing tests using the wrong domain. Using the wrong domain in the galley code, not the test directly, was causing errors that were annoying to debug. * docs. * Changelog. * docs. * sanitize-pr * Fixup * FS-1179: Moving a delete call into Brig, and calling it from Galley. Moving the deletion of one-on-one conversations into Brig, as Galley can't access the tables where that info is stored. This added a new internal API endpoint. Integration tests are currently failing on trying to delete one-on-one connections in Brig. * FS-1179: Fixing tests by telling cassandra to run queries anyway * Update docs/src/understand/configure-federation.md Co-authored-by: Sven Tennie <sven.tennie@gmail.com> * Update docs/src/developer/developer/federation-design-aspects.md Co-authored-by: Sven Tennie <sven.tennie@gmail.com> * FS-1179: Coping code from other tests to setup remote conversations. Setitng up remote conversations, using code from another test. Querying this database status requires messing around with Cql and polysemy rather than being able to look at API requests and reponses. The test is currently broken and I need to dig into why, and it is possible that this is a timing issue between brig and galley, but I'm not certain. * Fix cannon config map * whitespace * FS-1179: Improving tests in galley, and adding a new test to brig * FS-1179: PR formatting * Adding a round-trip test for rabbitmq. Using the pub and sub functions from the main galley code * hi ci * docs. * docs. * FS-1179: Add an internal endpoint for deleting a federation domain * FS-1179: Adding a defederation worker to background worker Adding a new worker on background worker that reads from a queue and calls Galley to delete a federation domain. * FS-1179: Some timeout config for the background worker. * Keeping brig and backend worker in sync * WIP: New tests and various changes to background worker * hi ci * Tweak docs. * nit-pick. * Mark test case as flaky. * Tweak log levels. * Tweak log msg. * wip * Update libs/wire-api/src/Wire/API/Routes/Internal/Brig.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update docs/src/understand/configure-federation.md Co-authored-by: fisx <mf@zerobuzz.net> * Update services/brig/src/Brig/API/Internal.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update services/brig/src/Brig/API/Internal.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update services/brig/src/Brig/API/Internal.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update services/federator/src/Federator/Run.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update services/galley/src/Galley/App.hs Co-authored-by: fisx <mf@zerobuzz.net> * FS-1115: PR notes * try fix helm charts * WIP * hi ci * changed federation strategy in CI from allowDynamic to allowAll * correct error in case of allowDynamic * default case made explicit * FS-1179: Modifications to tests to help stop hangups * Cleaning up comments * Create use domain names in integration tests. (this avoids hypothetical concurrency problems.) * Better errors in /integration. * nit-picks. * Better errors in /integration. * revert commit noise. * Fix `make list-flaky-tests`. * nit-picks, renames, minor refactorings. * s/type/newtype/ * Polish Wire.API.FederationUpdate (names and declaration order). * Fix: Always cancel `syncFedDomainUpdateThread`. * hi ci * Fixing things post-merge * Adding a changelog entry * FS-1179: Brig now deletes the notification queue for deleted domains * FS-1179: Removing and errant import * FS-1179: PR sanitisation. * FS-1179: Updating a TODO * FS-1179: Writing cleanup code so that we don't have dangling threads * FS-1179: PR formatting * PR formatting * More comments and exception handling * Fixing tests now that they have moved to hspec * Formatting * Fix background-worker integration tests. * Updating the internal notes * Removing a redundant language extension * FS-1179: Reworking after a discussion with Akshay about how to use AMQP. Workers are going to have their own connection management to RabbitMQ which will keep things simpler and help reduce potential blocks in processing. It also helps us not step on each other's toes when writing this service. * PR formatting * Hi CI * FS-1179: Adding galley and brig to background-worker's configmap * FS-1179: Removing the confusing "integration" tests for background-worker They were never actual integration tests, I was just confused about why `make ci` wasn't picking up the existing tests, so I made it run them and that was a mistake and caused confusion. * PR notes * `make sanitize-pr`. * Fixup * Sanitizing code * FS-1179: Removing dead code and bumping schema migrations * FS-1179: Removing more dead code * PR formatting * Hi CI * Fixing linter errors --------- Co-authored-by: Matthias Fischmann <mf@zerobuzz.net> Co-authored-by: Leif Battermann <leif.battermann@wire.com> Co-authored-by: Stefan Matting <smatting@users.noreply.github.com> Co-authored-by: Sven Tennie <sven.tennie@gmail.com>
* Servantify: `get "/i/users" (continue listAccountsByIdentityH)` * Servantify: `get "/i/users" (continue listActivatedAccountsH)` * Servantify: misc brig internal api - `get "/i/users/:uid/contacts" (continue getContactListH)` - `get "/i/users/activation-code" (continue getActivationCodeH)` - `get "/i/users/password-reset-code" (continue getPasswordResetCodeH)` * Servantify: misc brig internal api - `post "/i/users/revoke-identity" (continue revokeIdentityH)` - `head "/i/users/blacklist" (continue checkBlacklistH)` - `delete "/i/users/blacklist" (continue deleteFromBlacklistH)` - `post "/i/users/blacklist" (continue addBlacklistH)` - `get "/i/users/phone-prefixes/:prefix" (continue getPhonePrefixesH)` - `delete "/i/users/phone-prefixes/:prefix" (continue deleteFromPhonePrefixH)` - `post "/i/users/phone-prefixes" (continue addPhonePrefixH)` - `put "/i/users/:uid/sso-id" (continue updateSSOIdH)` - `delete "/i/users/:uid/sso-id" (continue deleteSSOIdH)` - `put "/i/users/:uid/managed-by" (continue updateManagedByH)` - `put "/i/users/:uid/rich-info" (continue updateRichInfoH)` - `put "/i/users/:uid/handle" (continue updateHandleH)` - `put "/i/users/:uid/name" (continue updateUserNameH)` - `get "/i/users/:uid/rich-info" (continue getRichInfoH)` - `get "/i/users/rich-info" (continue getRichInfoMultiH)` - `head "/i/users/handles/:handle" (continue checkHandleInternalH)` * Servantify: misc brig internal api - `put "/i/connections/connection-update" (continue updateConnectionInternalH)` * Servantify: misc brig internal api - `post "/i/clients" (continue internalListClientsH)` - `post "/i/clients/full" (continue internalListFullClientsH)` - `post "/i/clients/:uid" (continue addClientInternalH)` - `post "/i/clients/legalhold/:uid/request" (continue legalHoldClientRequestedH)` - `delete "/i/clients/legalhold/:uid" (continue removeLegalHoldClientH)` Co-authored-by: Paolo Capriotti <paolo@capriotti.io>
Stern: Batch=100
Co-authored-by: Stefan Matting <stefan@wire.com>
this flag is under controller.admissionWebhooks, not controller.service.admissionWebhooks. Fix yaml indentation.
* Breadcrumbs. * Better breadcrumbs. We don't know which release this will land in yet, PR is more future-proof. * Polish docs. * afterthought. * Update services/brig/test/integration/API/Federation.hs * More tests. * Remove a lying comment. * Refactor: keep the semantics of `_runSettings` intact. (This is part of the `Opts` data structure read from a file. it's very confusing to start updating part of that on the fly, and ignoring whatever comes from the config file.) * Remove a lying comment. * Fixup / WIP * Fixup * I take it back! * more docs. * rm bogus TODO, add a non-bogus one :) * revert changes of status code in integration tests. (Double-check if we can't keep the prod code behavior intact to avoid upgrade / fed-to-fed api / client api issues.) * Various updates, mainly moving some common code into wire-api and minimising types that are used in several common paths. * FS-1115: More common code for federation domain updates * Use tinylog instead of `print`. * sanitize-pr * Use retry instead of threadDelay-loop. * FS-1115: Updating design docs on how update intervals are supplied * Clarify source comment. * FS-1115: Cleaning up un-needed changes to options * update docs.wire.com * docs. * docs. * Fix: `/i/user/meta-info` (stern) (#3281) * Handle race conditions in /integration (#3278) * Update docs.wire.com (#3284) * Deprecate start-services-only.sh * Fix outdated docs on how to run swagger docs locally. * Restore deleted scripts (#3287) * Refactor federation domain configuration. - move strategy config from federator to brig. - remove domain list from strategy type. - add 'none' strategy (more explicit that 'list' with empty list, but both still work) * Fixup * Complete CRUD api for federator remotes (Update is missing) [WIP]. * Complete CRUD api for federator remotes (Update is missing): tests. * docs. * Cleanup * FS-1179: Setting up a location in galley for processing deleted domains NOTE: Broken! Currently not compiling Adding a new query to cassandra for getting remote members and conversation IDs that they are in, setting up loops for removing these members and potentially conversations using existing APIs. * FS-1179: Fixing the polysemy error. Picking an error type and documenting what might need to be done for better error mapping. * FS-1179: Initial processing for deleting remote domains. Setting up Galley to remove remote users from local conversations, local users from remote conversations, and to delete all user connections for the federation domain that is being dropped. * FS-1179: Fixing an error http reponse code. * wip * Remove dead code. * nit-pick. * Typo. * Fine-tune logging. * Fix haddocks. * Implement put. * Clariy updateFrequency everywhere. * nit-pick. * Fixup * Cleanup * Implement put. (For real this time.) * Move integration tests to /integration. * FS-1179: Work in progress. Tests are broken due to timing issues * FS-1179: Fixing an errant delete * FS-1179: Commiting and switching to FS-1115 * ... * wip * ... * FS-1115: Setting up calls to Brig from integration tests. JSON, calling, etc. * ... * ... * ... * ... * ... * ... * Fix * ... * WIP: Chasing down cache issues between some bug fixes * ... * ... * ... * ... * Tests are passing! * process leif's feedback. * ... * Fixing compile issues after a merge * Updating templates * Removing a JSON roundtrip, better using aeson. * Moving more of the federation domain update code into wire-api. Tests are broken with these changes. * wip * sanitize-pr * FS-1115: Updating brig integration config to help tests * s/AllowList/AllowDynamic/g * rm trailing whitespace. * docs * Updating with PR feedback * Have federation domain tests use MakesValue more * FS-1115: Removing more FedConn type specific code from federation tests * More code leaning on typeclasses * Pre-emptive rework before merging * Mark flaky test case. * docs. * Formatting * FS-1179: Setting up the receiving RabbitMQ loop. Setup a reading loop for RabbitMQ to get the domains that need to be deleted. We're using a queue here so that we have a non-volatile store for these domains that can automatically handle galley instances dying or being scaled down. * wip * FS-1179: Refactoring the RabbitMQ code Refactoring RabbitMQ code so that there is a single channel for both production and consumption of messages. There is also a function for "simple" processing where a message queuing system isn't available, but it still needs to have persistence and task co-ordination defined for it. This may not be a problem if we decide to mandate the use of RabbitMQ, or a similar queue. * Code formatting * FS-1179: Fixing tests using the wrong domain. Using the wrong domain in the galley code, not the test directly, was causing errors that were annoying to debug. * docs. * Changelog. * docs. * sanitize-pr * Fixup * FS-1179: Moving a delete call into Brig, and calling it from Galley. Moving the deletion of one-on-one conversations into Brig, as Galley can't access the tables where that info is stored. This added a new internal API endpoint. Integration tests are currently failing on trying to delete one-on-one connections in Brig. * FS-1179: Fixing tests by telling cassandra to run queries anyway * Update docs/src/understand/configure-federation.md Co-authored-by: Sven Tennie <sven.tennie@gmail.com> * Update docs/src/developer/developer/federation-design-aspects.md Co-authored-by: Sven Tennie <sven.tennie@gmail.com> * FS-1179: Coping code from other tests to setup remote conversations. Setitng up remote conversations, using code from another test. Querying this database status requires messing around with Cql and polysemy rather than being able to look at API requests and reponses. The test is currently broken and I need to dig into why, and it is possible that this is a timing issue between brig and galley, but I'm not certain. * Fix cannon config map * whitespace * FS-1179: Improving tests in galley, and adding a new test to brig * FS-1179: PR formatting * Adding a round-trip test for rabbitmq. Using the pub and sub functions from the main galley code * hi ci * docs. * docs. * FS-1179: Add an internal endpoint for deleting a federation domain * FS-1179: Adding a defederation worker to background worker Adding a new worker on background worker that reads from a queue and calls Galley to delete a federation domain. * FS-1179: Some timeout config for the background worker. * Keeping brig and backend worker in sync * WIP: New tests and various changes to background worker * hi ci * Tweak docs. * nit-pick. * Mark test case as flaky. * Tweak log levels. * Tweak log msg. * wip * Update libs/wire-api/src/Wire/API/Routes/Internal/Brig.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update docs/src/understand/configure-federation.md Co-authored-by: fisx <mf@zerobuzz.net> * Update services/brig/src/Brig/API/Internal.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update services/brig/src/Brig/API/Internal.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update services/brig/src/Brig/API/Internal.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update services/federator/src/Federator/Run.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update services/galley/src/Galley/App.hs Co-authored-by: fisx <mf@zerobuzz.net> * FS-1115: PR notes * try fix helm charts * WIP * hi ci * changed federation strategy in CI from allowDynamic to allowAll * correct error in case of allowDynamic * default case made explicit * FS-1179: Modifications to tests to help stop hangups * Cleaning up comments * Create use domain names in integration tests. (this avoids hypothetical concurrency problems.) * Better errors in /integration. * nit-picks. * Better errors in /integration. * revert commit noise. * Fix `make list-flaky-tests`. * nit-picks, renames, minor refactorings. * s/type/newtype/ * Polish Wire.API.FederationUpdate (names and declaration order). * Fix: Always cancel `syncFedDomainUpdateThread`. * hi ci * Fixing things post-merge * Adding a changelog entry * FS-1179: Brig now deletes the notification queue for deleted domains * FS-1179: Removing and errant import * FS-1179: PR sanitisation. * FS-1179: Updating a TODO * FS-1179: Writing cleanup code so that we don't have dangling threads * FS-1179: PR formatting * PR formatting * More comments and exception handling * Fixing tests now that they have moved to hspec * Formatting * Fix background-worker integration tests. * Updating the internal notes * WPB-240: Initial implementation of notifications sending for defederation * Removing a redundant language extension * Removing a redundant language extension * WPB-240: Matching event fanout count to the limit in options. Adding tests for event fanout. * WPB-240: Comments and code in tests for quickly filling the DB * FS-1179: Reworking after a discussion with Akshay about how to use AMQP. Workers are going to have their own connection management to RabbitMQ which will keep things simpler and help reduce potential blocks in processing. It also helps us not step on each other's toes when writing this service. * PR formatting * Hi CI * FS-1179: Adding galley and brig to background-worker's configmap * FS-1179: Removing the confusing "integration" tests for background-worker They were never actual integration tests, I was just confused about why `make ci` wasn't picking up the existing tests, so I made it run them and that was a mistake and caused confusion. * PR notes * `make sanitize-pr`. * Fixup * Changing the format of federation.delete. * Sanitizing code * FS-1179: Removing dead code and bumping schema migrations * FS-1179: Removing more dead code * PR formatting * Hi CI * WPB-240: Formatting code * WPB-240: Changelog entries * WPB-240: Removing dead code * WPB-240: Moving notification code into an effect, as mentioned in PR. --------- Co-authored-by: Matthias Fischmann <mf@zerobuzz.net> Co-authored-by: Leif Battermann <leif.battermann@wire.com> Co-authored-by: Stefan Matting <smatting@users.noreply.github.com> Co-authored-by: Sven Tennie <sven.tennie@gmail.com>
SSO Faq entry on CSP.
We wish to avoid needing persistence during CI runs, as we run into limits of volume creation, which causes integration test setup failures. This PR can be reverted at some point in the future when: - hedis code is both compatible with current redis version AND redis version 7 and its tests pass, and it is released to hackage / nixpkgs. Then: - Step 1. We can use latest hedis code (and merge this to develop, ensuring this works on staging) - Step 2: Subsequently this PR can be reverted, and the redis-cluster helm chart bumped, and persistence disabled. See also #3444 - ensure to do steps 1 and 2 on *separate* PRs, to avoid requiring a redis helm chart update concurrently with a code update - that would not work well in production and cause downtime. Context: - https://wearezeta.atlassian.net/browse/WPB-2824 - https://wearezeta.atlassian.net/browse/WPB-2783 - #3427 - #3444 - #3445
* federator-external: Add status endpoint, migrate to servant * charts/federator: Use status endpoint for liveness and readiness probes * Extract Endpath into the extended library * federator: Extract 'RPC' type in its own module * federator-internal: Add status endpoint, migrate to servant * federator: Make /i/status from any port reflect status of both ports
Co-authored-by: Akshay Mankar <itsakshaymankar@gmail.com>
|
|
zebot
added
the
ok-to-test
battermann
force-pushed
the
bat/merge-dev-into-mls
branch
from
f177c69
to
4832f50
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist
changelog.d