[BEAM-9324] Fix incompatibility of direct runner with cython

[Abacn]

This is due to Cython.Shadow.cast(Receiver, value) attempting to run Receiver(value) (probably very old code in Python 2 age where "old" class did not have __call__)

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[asf-ci]

Can one of the admins verify this patch?

[asf-ci]

Can one of the admins verify this patch?

[asf-ci]

Can one of the admins verify this patch?

[Abacn]

Run Python PreCommit

[codecov]

Codecov Report

Merging #17728 (f7f142f) into master (bac4e28) will decrease coverage by 0.00%.
The diff coverage is 76.47%.

@@            Coverage Diff             @@
##           master   #17728      +/-   ##
==========================================
- Coverage   73.99%   73.99%   -0.01%     
==========================================
  Files         696      696              
  Lines       91851    91857       +6     
==========================================
+ Hits        67964    67967       +3     
- Misses      22638    22641       +3     
  Partials     1249     1249              

Flag	Coverage Δ
python	83.73% <76.47%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ks/python/apache_beam/runners/worker/operations.py	74.02% <76.47%> (-0.07%)	⬇️
sdks/python/apache_beam/runners/direct/executor.py	96.46% <0.00%> (-0.55%)	⬇️
sdks/python/apache_beam/transforms/combiners.py	93.05% <0.00%> (-0.39%)	⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py	88.94% <0.00%> (-0.16%)	⬇️
sdks/python/apache_beam/transforms/core.py	92.30% <0.00%> (ø)
sdks/python/apache_beam/io/gcp/bigquery.py	65.10% <0.00%> (+0.11%)	⬆️
...hon/apache_beam/runners/worker/bundle_processor.py	93.68% <0.00%> (+0.24%)	⬆️
.../apache_beam/runners/interactive/dataproc/types.py	100.00% <0.00%> (+4.16%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bac4e28...f7f142f. Read the comment docs.

[Abacn]

R: @TheNeuralBit
R: @robertwb

[TheNeuralBit]

I've seen the errors from BEAM-9324 TypeError: Receiver() takes no argument before when I've got my build into an inconsistent state (some files Cythonized, some not). I wonder if there's a build system change we can make to help avoid this situation rather than branching throughout the code?

[Abacn]

This was a bug of cython Shadow fixed in upcoming v3.0 (cython/cython@3149eb8). Before cython 3.0 is released this patch looks needed.

In the translated c-code the helper function gives

  /* "apache_beam/runners/worker/operations.py":95
 * 
 * def _cast_to_operation(value):
 *   if cython.compiled:             # <<<<<<<<<<<<<<
 *     return cython.cast(Operation, value)
 *   else:
 */
  __pyx_t_1 = (1 != 0);
  if (__pyx_t_1) {

...

    goto __pyx_L0;

/* ... */
  }

  /* ... */
  /*else*/ {
    ...
    goto __pyx_L0;
  }

  /* ... */

  /* function exit code */
  ...
  __pyx_L0:;
 ...
  return __pyx_r;
}

where the branching condition is constant in the compile time and should be eliminated by c compiler.

[TheNeuralBit]

Ah ok, one last question from me: should we remove this when/if cython 3 is out and we upgrade it? If so let's file a jira and maybe add TODOs.

[Abacn]

Ah ok, one last question from me: should we remove this when/if cython 3 is out and we upgrade it? If so let's file a jira and maybe add TODOs.

Added TODO. used this jira for reference since Cython3 may still be pending in nearly future and thus not filing a ticket at this moment.

[TheNeuralBit]

Run Python PreCommit

[TheNeuralBit]

Run PythonDocker PreCommit

[TheNeuralBit]

Thanks!