Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BEAM-22] Support Unbounded PCollections in same-process execution #3

Closed
wants to merge 3 commits into from
Closed

[BEAM-22] Support Unbounded PCollections in same-process execution #3

wants to merge 3 commits into from

Commits on Mar 1, 2016

  1. Implement InProcessEvaluationContext 

    This is the primary "global state" object for the evaluation of a
Pipeline using the InProcessPipelineRunner, and is responsible for
properly routing information about the state of the pipeline to
transform evaluators.

Remove the InProcessEvaluationContext from the InProcessPipelineRunner
class, and implement as a class directly. Fix associated imports.
    @tgroh
    tgroh committed Mar 1, 2016
    Configuration menu
    Copy the full SHA
    f5e8789 View commit details
    Browse the repository at this point in the history

  2. Add ExecutorServiceParallelExecutor as an InProcessExecutor 

    This is responsible for scheduling transform evaluations and
communicating results back to the evaluation context. The executor
handle PTransforms that block arbitarily waiting for additional input.
    @tgroh
    tgroh committed Mar 1, 2016
    Configuration menu
    Copy the full SHA
    b953f26 View commit details
    Browse the repository at this point in the history

  3. Implement InProcessPipelineRunner#run 

    Appropriately construct an evaluation context and executor, and start
the pipeline when run is called.

Implement runner-provided ExecutionContext and StepContext abstractions.
    @tgroh
    tgroh committed Mar 1, 2016
    Configuration menu
    Copy the full SHA
    ccc368a View commit details
    Browse the repository at this point in the history