[BEAM-14422] Adding exception testing for ReadFromBigQuery and WriteToBigQuery #17589
Conversation
|
Can one of the admins verify this patch? |
4 similar comments
|
Can one of the admins verify this patch? |
|
Can one of the admins verify this patch? |
|
Can one of the admins verify this patch? |
|
Can one of the admins verify this patch? |
ahmedabu98
changed the title
| query='SELECT * FROM `project.dataset.table`', | ||
| gcs_location='gs://temp_location') | ||
|
|
||
| self.assertEqual(16, mock_api.call_count) |
There was a problem hiding this comment.
We expect 16 calls here because get_or_create_dataset (where the dataset Insert API is called from) has the @retry... decorator and create_temporary_dataset, where get_or_create_dataset is called from, also has the @retry... decorator.
For each create_temporary_dataset retry, we have four get_or_create_dataset retries, giving us a total of 16.
Having 16 retries seems weird, but @pabloem and I thought this might be by design to emulate an exponential backoff and may be of benefit to users. Thoughts on this?
|
R: @pabloem |
|
Run Portable_Python PreCommit |
1 similar comment
|
Run Portable_Python PreCommit |
|
trying a fix myself : ) |
| exception_type=exceptions.ServiceUnavailable, | ||
| error_message='backendError'), | ||
| ]) | ||
| @mock.patch('time.sleep') |
There was a problem hiding this comment.
also try moving this into a context manager?
|
Run Python PreCommit |
|
@ahmedabu98 I gave up on figuring out the env difference and change the assertion to just check |
|
Run Python 3.8 PostCommit |
Codecov Report
@@ Coverage Diff @@
## master #17589 +/- ##
==========================================
+ Coverage 73.86% 73.95% +0.08%
==========================================
Files 690 695 +5
Lines 91211 91883 +672
==========================================
+ Hits 67374 67948 +574
- Misses 22605 22703 +98
Partials 1232 1232
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
|
Run Python PreCommit |
|
Run Python 3.8 PostCommit |
I think |
|
lgtm thanks! |
|
Potentially release blocking for 2.40.0. (/cc @tvalentyn @pabloem ) |
…or ReadFromBigQuery * create temp dataset exception test * working with real dataset and table, need to make it work with dummy * error message is correct * temp dataset exception done * query exception test + some fixes * export read job exception * exception test for load job fail * trying fix * copy load job exception draft * using context managers instead of decorators for patch in read exception tests * using context manager for file loads exceptions * test for copy load job exceptions * ignore * make create temp dataset exception faster * reducing number of exceptions for copy load job * ignore * testing less exception types to reduce test speed * fix formatter * fix lint * fixup Co-authored-by: Pablo E <pabloem@apache.org> Co-authored-by: Pablo <pabloem@users.noreply.github.com>
ahmedabu98
changed the title
No description provided.