Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[EventHubs] merge pyamqp to main (#27763)
* [EventHub] basic receive event scenario with pyamqp (#19748) * initial changes for receiving * undo __init__ aio * vendor * remove c/pyx files * adams comments * [EventHubs & AMQP Python] Send Port (#19745) * draft send port * copy and paste code changes in amqp * simple stress test scripts for sending and receiving * review feedbacks * [EventHubs&AMQP Python] Port amqp send large message (#19937) * port amqp send large message * send perf test in parallel * use context manager for executor * add throughput * improve test code * update test matrix * update test * handle server busy * fix timeout * precision to 2 decimal points * update pyamqp changes and update tests * [EventHubs] Pure Python AMQP Sycn Implementation Integration (#22397) * copy amqp changes * eh python amqp integration * fix time unit * rename module pyamqp to _pyamqp * more pyamqp to _pyamqp * simplify todo * [EH Pyproto] Release preparation (#22433) * cherry pick changes * update docs * cherry pick fixed retry PR * minor fix * fix mypy, pylint, brokenlink * update doc * opt out mypy/pylint/api stub * try opt out checkpointstore in ci and test * fix * more fixes * furuther opt out tests * update tests * bump version * fix __str__ * add test play holder * ignore azure checkpoinstore aio in ci Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com> * update readme to drop uamqp * revert async tests * revert aio module in eventhub * [EH Pyproto] Async support (#22957) * async port * add scripts for tests * update async perf test scripts * fix test scripts * amqp implementation update + eh update + sync perf test scripts * update pyamqp and eh async impl and test scripts * update pyamqp async impl * fix bug * fix pyamqp transport ssl setting and asyncio exception module import * use ensure future for 3.6 * update token generation to return bytes to avoid breaking changes * update docs * Increment version for eventhub releases (#22994) Increment package version after release of azure-eventhub * [EH Pyproto] Async recv perf improvement (#23122) * stop spawning too much coroutines * improve send * async recv perf improvement * async perf improve * update version * align with sync imple * update method name * remove redundant except catch * [EH Pyproto] Release updates (#23349) * update docs * add todo * Increment version for eventhub releases (#23420) Increment package version after release of azure-eventhub * AMQP websocket implementation (#23722) * Initial implementation * http proxy support * change impl * more changes * working sol * async impl * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/_transport.py * more changes * sasl mixin * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/message.py * refactor * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/_transport.py * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/_transport.py * oops * comments * comment * Apply suggestions from code review Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com> * comments * changes * async test * rasie * lint * changelog * version * comments * move path to EH Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com> * Revert "AMQP websocket implementation (#23722)" (#24344) This reverts commit 0123f4d. * AMQP websocket implementation (#24345) * Initial implementation * http proxy support * change impl * more changes * working sol * async impl * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/_transport.py * more changes * sasl mixin * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/message.py * refactor * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/_transport.py * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/_transport.py * oops * comments * comment * Apply suggestions from code review Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com> * comments * changes * async test * rasie * lint * changelog * version * comments * move path to EH * Fix typo Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com> * [EventHub] basic receive event scenario with pyamqp (#19748) * initial changes for receiving * undo __init__ aio * vendor * remove c/pyx files * adams comments * [EventHubs & AMQP Python] Send Port (#19745) * draft send port * copy and paste code changes in amqp * simple stress test scripts for sending and receiving * review feedbacks * [EventHubs&AMQP Python] Port amqp send large message (#19937) * port amqp send large message * send perf test in parallel * use context manager for executor * add throughput * improve test code * update test matrix * update test * handle server busy * fix timeout * precision to 2 decimal points * update pyamqp changes and update tests * [EventHubs] Pure Python AMQP Sycn Implementation Integration (#22397) * copy amqp changes * eh python amqp integration * fix time unit * rename module pyamqp to _pyamqp * more pyamqp to _pyamqp * simplify todo * [EH Pyproto] Release preparation (#22433) * cherry pick changes * update docs * cherry pick fixed retry PR * minor fix * fix mypy, pylint, brokenlink * update doc * opt out mypy/pylint/api stub * try opt out checkpointstore in ci and test * fix * more fixes * furuther opt out tests * update tests * bump version * fix __str__ * add test play holder * ignore azure checkpoinstore aio in ci Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com> * update readme to drop uamqp * revert async tests * revert aio module in eventhub * [EH Pyproto] Async support (#22957) * async port * add scripts for tests * update async perf test scripts * fix test scripts * amqp implementation update + eh update + sync perf test scripts * update pyamqp and eh async impl and test scripts * update pyamqp async impl * fix bug * fix pyamqp transport ssl setting and asyncio exception module import * use ensure future for 3.6 * update token generation to return bytes to avoid breaking changes * update docs * Increment version for eventhub releases (#22994) Increment package version after release of azure-eventhub * [EH Pyproto] Async recv perf improvement (#23122) * stop spawning too much coroutines * improve send * async recv perf improvement * async perf improve * update version * align with sync imple * update method name * remove redundant except catch * [EH Pyproto] Release updates (#23349) * update docs * add todo * Increment version for eventhub releases (#23420) Increment package version after release of azure-eventhub * AMQP websocket implementation (#23722) * Initial implementation * http proxy support * change impl * more changes * working sol * async impl * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/_transport.py * more changes * sasl mixin * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/message.py * refactor * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/_transport.py * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/_transport.py * oops * comments * comment * Apply suggestions from code review Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com> * comments * changes * async test * rasie * lint * changelog * version * comments * move path to EH Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com> * Revert "AMQP websocket implementation (#23722)" (#24344) This reverts commit 0123f4d. * AMQP websocket implementation (#24345) * Initial implementation * http proxy support * change impl * more changes * working sol * async impl * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/_transport.py * more changes * sasl mixin * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/message.py * refactor * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/_transport.py * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/_transport.py * oops * comments * comment * Apply suggestions from code review Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com> * comments * changes * async test * rasie * lint * changelog * version * comments * move path to EH * Fix typo Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com> * remove extra SR related code * update docs + type hints * fixing failing tests * [EventHubs] merge working websocket changes to feature branch (#24444) * adam's working changes * Adding back Rakshith's websocket changes (#24410) * Adding back Rakshith's sync websocket changes * fix async send and receive * fix transport bugs * add websocket to dev reqs + async fix hostname * thank you kashif * fix tests + turn on websocket tests * update consumer test timing * fix merge bugs + remove pyamqp specific tests * update sleep time in test * enable live test for ws receive * fix to create Batch properly * [eventhub] Websocket timeout error exception thrown (#24504) * sync websocket timeout to operationtimeout, not changing other transport types just yet * async websocket to operation timeout * upstream * default timeout 1.0 * default timeout 1 * throwing a socket timeout, operationtimeout was throwing out a real error * replacing socket.timeout with TimeoutError, added into except statments as well * timeout is inherited from oserror, dont need both * test timeoutexception throwing errors * deafult timeout to 1 * [eventhub] websocket default timeout fix (#24565) * websocket timeout fix * timeout interval for both ssl and webscoket * [eventhub] Custom Endpoint (#24505) * sync ce * async ce * add string ending * only pass to transport * running into same recieve issue with sync * fixing async - needs to pass to sasl * remove logger * stopping here * adding prefix to fix sample * add in prefetch * fixing transport remove print * host being overriden * removing trace * fix to use url async * aligning sync/async pattern * removing uneeded hostname switch * string formatting * changelog * adding docstrings for supported events * pr comments refactoring sync * mirroring on async * pr comment docstring * removing import * missing _ * missing ssl * if no port given, we use default set in config * async of same ^ * add default port in connection stage if port is none * adding in docstring to cliet/connection string constructor * custom_endpoint_address in client base async to match sync * fix import on websocket test * fix import 2 * skipping tests * removing import * pytest.mark * [EventHubs] pyproto - update release date + docs (#24723) * add async doc rst file * Increment version for eventhub releases (#24753) Increment package version after release of azure-eventhub * changes to update status (#25024) * updating codeowners file in pyproto feature * [Eventhub] pyamqp prefetch fix (#24890) * prefetch fix * adding async - sorry! * async * Use --no-cone in pipeline sparse checkout script (#25165) (#25208) Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com> Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com> Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com> * [AMQP Python] Eventhub Pyamqp tests (#24895) * starting tests * updates to websocket sync * moving around format - unittest and live test * live test + unittests starting * websocket async passing * eol * assert not return * assert not return * fixed assert * auth tests * auth unittest pyamqp * replicating uamqp tests * keep_alive_thread * skip for now - no keep alive * pickle/deepcopy, might not want to keep all * stopping here for now - need tls on rabbitmq * cleaning up tests - pickle * removing and editing uneeded tests * removing unused test * added receive amqp tests * exceptions with pytest.raises, not live * moving around tests * testing mgmt calls like _start_producer * Use --no-cone in pipeline sparse checkout script (#25165) Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com> * unused imports Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com> Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com> * [Pyamqp] Fix network logging trace in client_base (#25218) * pass in right kwarg for network tracing * remove client changes. Another PR * reverting link credit for now (#25310) * [Pyamqp] Pyampq debug build Linkedin (#25296) * enhanced logging for linkedin * stuff * fixes * minor sample fix * Changelog * remove unused imports * fix formatting changes * change debug level * update version info * update changelog * fix sample * fix logging message for empty access token * Increment version for eventhub releases (#25320) Increment package version after release of azure-eventhub * removing duplicate (#25321) * add async unit tests (#25396) * reset logging level (#25588) * [Pyamqp] Remember Proxy Params (#25564) * fix to keep proxy params * async changes for proxy * unit tests * changes * more changes Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com> Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com> * [Pyamqp] Intial TODOS Clean Up (#25630) * set default SSL version * dont need it for EH & SB * wont impact us, check mtg notes * decode error wont affect us * address in SB PR * change language on logger * it does close socket * no other closes needed * keep for better tracking * close w/ error when max frame size is invalid * detach links on session end * clean up links on session outgoing_end * reject link by detaching * uncomment logging for later review * reject the link that was set in the try * [Pyamqp] test fixes for pipeline (#25749) * test fixes for pipeline * connect to EHerror * [PyAMQP] Stress testing reform (#25770) * reform stress test * reform stress test * updating stress test format * can specify azure_identity on producer * epoch level sync * removing commands * aligning sync/async * pyamqp logging * uamqp to pyamqp * import remove * fixing deploy commands * values file * checking eng/ file resources * remove version * moving location of this folder * imagepullpolicy, azure_identity for consumer * [PyAMQP] Updating pyamqp with SB changes (#25804) * pyamqp diff from anna's branch * mgmt_request returns code, response, desc * mgmt_request returns code, response, desc async * handle error being thrown * handle error being thrown 2 * handle error being thrown 3 * Fixed error path * fix error path Co-authored-by: Anna Tisch <antisch@microsoft.com> Co-authored-by: Kashif Khan <361477+kashifkhan@users.noreply.github.com> * removing buff producer to add back in later * update cspell * fix uppercase link in doc/dev/issues/resolve_issues_effectively * [PyAMQP] Connections TODO (#26018) * protocol errors * fix formatting * [PyAMQP] Kashif Client refactoring changes (#25451) Linked to link #22051 for a TODO about typing in docstrings * need to use pyamqp (#25895) * [EventHubs] add amqp switch support (#25965) This PR is for adding switch support to the `feature/eventhub/pyproto` changes including the uamqp switch from current `main`. fixes #21246 Addressing Anna's comments from uamqp switch PR (#25193) + main changes: - [x] Moving pyamqp logic out to the PyamqpTransport - [x] **Confirmed: The size of encoded pyamqp.Message is larger than uamqp.Message.** I thought otherwise b/c I was adding the header/property objects by default when building the outgoing uamqp message, even if all values inside those are None. I've fixed this. - [ ] Make BatchMessage transport agnostic: #25494 (comment) - Instead, updated EventDataBatch so that it takes an amqp_transport. If EventDataBatch is manually created and uses PyamqpTransport, inside `send_batch()`, if the producer client transport uses UamqpTransport, the BatchMessage corresponding to the client's amqp_transport will be built and sent. - [X] Add `message` property to `EventData`/`EventDataBatch`, which return `LegacyMessage`/`LegacyMessageBatch` from `_pyamqp` for backcompat. - [x] add `connection=None` parameter to `pyamqp.AMQPClient.open()` as per [this discussion](#25494 (comment)) - [x] Add an async SharedConnectionManager in pyamqp: #25494 (comment) Issue created [[here](#25875)] to address the below TODOs in a separate PR: - [x] add TODO in pyamqp that SenderClient should take msg_timeout: #25494 (comment) - [x] add TODO in pyamqp that ReceiveClient should take timeout: #25494 (comment) - [x] add both MAX_MESSAGE_LENGTH_BYTES and MAX_FRAME_SIZE_BYTES to pyamqp: #25494 (comment) TODO: - [ ] fix mypy/pylint issues - made partial progress. fix rest in separate PR for issue: #25936 - [x] investigate how to remove `message` property from public API. - mark as deprecated and log a deprecation warning if accessed * prep release * [Pyamqp] Exception Todos (#25893) * throw proper exception and error condition on CBS * error condition for ready * close when open is on non-zero channel * error condition is proper for timeout * unattached handle * max handles error condition * fix strings * fix * end the session on unattached handle * remove TODO * conditions are proper * raise link error on close or detach * revert change * remove TODO * change to client error * change to AmqpError * change to AMQPError * detach called * detach the link, dont close the connection * comments clean up + change condition * handle invalid channel on end * descriptive error message * fix formatting * detach err message on max handle * [Pyamqp] Async WS implementation using a pure async library (#26234) * init changes for aiohttp ws * fixes + formatting * fixes for context manager * changes * ssl options * clean up * move build opts in to mixin * pass in proxy information * attach port to proxy if given * address comments * remove self from proxy host * [SB PyAMQP] Servicebus PyAMQP Working off of Anna's Branch (#24975) * Added pyamqp * Added message compatibility tests * Start rewiring messages for pyamqp * Added message backcompat layer * Successful message send * Started receiver * Successful message receive * Message settlement * Fix other settlement outcomes * Make tests live * message partition_key if it can't be decoded - output value * removing references to __future___ annotations for now - not supported in 3.6 * comparing name of transport - not the object * passing in a dummy frame for new formatting of SBMessageReceived * adding in fake frame for message in queue tests * uamqp_mesage -> uamqp_message * state should be auth_state * switching this back - _message is Message * Improved typing * Revert "Improved typing" This reverts commit aeffcb2. * Fix TransportType enum * Fix import statement * Fix application property encoding * Skip queue iterator tests * Fix mgmt op timeout * Fixes to mgmt link * Fix frame decode tests * More mgmt fixes * Some message fixes * Fix session filters * Message tests * Skip more iterator tests * Update to retry policy * adding in support for websockets is CE supported? * fixing up pylint-still some issues * some more pylint/TODOs * pylint changes * fixing pylint * more pylint connection * More test fixes * Fix scheduling * Fix retry test * Fix error handling * Sender refactor for timeout * Fix link detach * Fixed receiver control flow * Update pyamqp async code * Updated sb async * Typing fix * Some async fixes * Skip async iter tests * Workaround socket timeout * Literal import * More async test fixes * Added keepalive * Pylint cleanup * fix mypy errors in _pyamqp * fix mypy sb layer * fix bug * unused import * lint * fix failing tests * ignore sb iterator receive samples Co-authored-by: antisch <antisch@microsoft.com> Co-authored-by: swathipil <swathip@microsoft.com> * [Pyamqp] Pyamqp fix conn (#26568) * remove unnecessary pops * fix var name + remove unnecessary pop * fix * Bring in changes to fix bandit from failing * [EventHubs&ServiceBus] merge sb and eh pyamqp (#26548) * merge sb and eh pyamqp * reenable pylint for EH * turn on mypy for EH * fix mypy errors eh layer * fix EH mypy/pylint * fix SB failing tests * fix more tests/mypy * import literal from typing extensions * remove whitespace * fix typing cast bug in EH * lint * fix port url in async transport * kashifs comments * mypy/lint/kashifs comments * [Pyamqp] Fix Async Invalid Host Error Test (#26595) * resert transport to original state * add in missing continue in except * Update sdk/eventhub/azure-eventhub/azure/eventhub/_pyamqp/aio/_transport_async.py Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com> Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com> * fix merge conflict stress * update cspell, ignore tables spelling * skip connection verify tests sb for now * fix logging formatting (#26682) * copy over kashifs change to sb * [ServiceBus&EventHubs] pyamqp - update EH/SB docs for release (#26741) updating docs to prepare for release * [ServiceBus&EventHubs] fix mypy/pylint (#26744) * fix mypy/pylint * bump SB version * prep release alpha (#26755) * Increment version for servicebus releases (#26762) Increment package version after release of azure-servicebus * Increment version for eventhub releases (#26766) Increment package version after release of azure-eventhub * [PyAMQP] Fix logging (#26785) * make logging network trace a debug level log * async changes * await async sleep (#26853) * fix for async socket (#26852) * [Pyamqp] Possible solutions for network disruption using async websocket (#26856) * heartbeat * constant value for heartbeat * address comments * [Pyamqp] Fix to Improve Websocket Sync and Async Network Disruption Handling (#27006) * changes * fix lint * remove prints * remove unused import * remove heartbeat from this PR * [pyAMQP] Stress fixes for aiohttp and valueError (#27034) * changes * fix lint * remove prints * remove unused import * remove heartbeat from this PR * OS Error to catch client closed session error * removing value error raise, change to logging -- causes error on network disconnect * when looping on open clinet * change to printf style Co-authored-by: Kashif Khan <kashifkhan@microsoft.com> * [Pyamqp] Changes for blocking exceptions (#27260) * changes for blocking exceptions * fix hang on unit test * lint fixes * fix bug for closes * fixes * lint issues * Stress testing updates (#27456) * can uncomment line 5 to run against git version of pyamqp * can move line 6 into scripts/dev_requirement file * test against newest version of pyamqp * increase test time, get rid of unused tests * removing test names * change naming * change naming * raise logging level to catch only error level * return logging to info * adding resource requests * message retention needs to last as long as the test * websocket async test * changing life of messages * 32 partitions * add uamqp flag - remove logging * adding before trying matrix * websocket dep * helm ignore * updating * updating tests * update consumer files * remove log lines * remove log lines * remove commented * update * [EventHubs] kwargs/error testing (#27065) * adding tests * add auth/connection tests + fixes * fix connection verify error handling * revert consumer retry change * call ws close in sync transport * typo * fix ws exc import * fix async transport * fix link detach vendor error exception parity * add operationtimeouterror * add more negative tests * annas comments + lint * lint + tests * add ids for uamqp vs pyamqp tests * update tests * skip macos tests * [EventHubs] check for any non-None values in amqp header/properties (#27444) * add any method to amqp header/properties * use count(None) to check non-None header/props vals * [EventHubs] update sync receive client ready flow (#27411) * update sync receive client ready flow * lint + mypy * fix reconnect test * [pyamqp] os error add (#27351) * os error add * update except statement * test mock of receive_bytes on pipeline * client os error * mock try 2 * pylint * fix patch line * fixed mock * newline * fixing spacing * unused import * fix error mssgs * changes from perf run (#27703) * Matrix Gen Stress Tests (#27754) * stress matrix gen * removing unused dockerfiles for now * revert sb to main * revert non-eh files * restore samples/readmes to main * restore ci/tests/shared reqs to main * update to stable * lint + fix tests for no uamqp import * update test timeout * re-organize changelog Co-authored-by: Adam Ling (MSFT) <adam_ling@outlook.com> Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com> Co-authored-by: Rakshith Bhyravabhotla <rakshith.bhyravabhotla@gmail.com> Co-authored-by: Kashif Khan <kashifkhan@microsoft.com> Co-authored-by: Libba Lawrence <llawrence@microsoft.com> Co-authored-by: Kashif Khan <361477+kashifkhan@users.noreply.github.com> Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com> Co-authored-by: Anna Tisch <antisch@microsoft.com>
- Loading branch information
9 people
committed
Dec 13, 2022
1 parent
a8041f5
commit d111b55