[BEAM-13314]Revise recommendations to manage Python pipeline dependencies. #16938

AnandInguva · 2022-02-24T18:43:04Z

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

AnandInguva · 2022-02-24T20:04:21Z

R: @tvalentyn

tvalentyn · 2022-03-04T20:33:26Z

website/www/site/content/en/documentation/runtime/environments.md

@@ -46,7 +46,7 @@ Beam [SDK container images](https://hub.docker.com/search?q=apache%2Fbeam&type=i

 1. **[Writing a new](#writing-new-dockerfiles) Dockerfile based on a released container image**. This is sufficient for simple additions to the image, such as adding artifacts or environment variables.
 2. **[Modifying](#modifying-dockerfiles) a source Dockerfile in [Beam](https://github.com/apache/beam)**. This method requires building from Beam source but allows for greater customization of the container (including replacement of artifacts or base OS/language versions).
-
+3. **[Build](#modify-existing-base-image) an existing container image to make it compatible with Apache Beam Runners**. This method is used when users start from an existing image, and configure the image to be compatible with Apache Beam Runners.


Suggested change

3. **[Build](#modify-existing-base-image) an existing container image to make it compatible with Apache Beam Runners**. This method is used when users start from an existing image, and configure the image to be compatible with Apache Beam Runners.

3. **[Modifying](#modify-existing-base-image) an existing container image to make it compatible with Apache Beam Runners**. This method is used when users start from an existing image, and configure the image to be compatible with Apache Beam Runners.

Also: one of three ways above.

tvalentyn · 2022-03-04T20:35:05Z

website/www/site/content/en/documentation/runtime/environments.md

+1. Copy necessary artifacts from Apache Beam base image to your image.
+  ```
+  # This can be any container image,
+ FROM python:3.8-slim


mismatch between py3.8 and py3.7 below

Thanks for catching

website/www/site/content/en/documentation/runtime/environments.md

tvalentyn · 2022-03-04T20:55:26Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

@@ -45,6 +45,19 @@ If your pipeline uses public packages from the [Python Package Index](https://py
    The runner will use the `requirements.txt` file to install your additional dependencies onto the remote workers.

 **Important:** Remote workers will install all packages listed in the `requirements.txt` file. Because of this, it's very important that you delete non-PyPI packages from the `requirements.txt` file, as stated in step 2. If you don't remove non-PyPI packages, the remote workers will fail when attempting to install packages from sources that are unknown to them.
+> **NOTE**: An alternative to `pip check` is to use a library like [pip-tools](https://github.com/jazzband/pip-tools) to compile the `requirements.txt` with all the dependencies required for the pipeline.


pip freeze, not pip check

you can explain:

...to compile the requirements.txt` all transitive dependencies from a smaller set of requirements.```

tvalentyn · 2022-03-04T20:56:51Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+       COPY <path to requirements.txt> /tmp/requirements.txt
+       RUN python -m pip download -r /tmp/requirements.txt
+
+**Note:** [Different approaches](https://beam.apache.org/documentation/runtime/environments/#writing-new-dockerfiles) to build the container images that would be compatible with Apache Beam Runners.


I don't think this is relevant here

I thought may be referencing on how to use custom container would be useful but thinking about it, you are right

tvalentyn · 2022-03-04T20:57:07Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+       # Add these lines with the path to the requirements.txt to the Dockerfile
+
+       COPY <path to requirements.txt> /tmp/requirements.txt
+       RUN python -m pip download -r /tmp/requirements.txt


why pip download and not pip install ?

tvalentyn · 2022-03-04T21:00:18Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+
+You can pass a [container](https://hub.docker.com/search?q=apache%2Fbeam&type=image) image with all the dependencies that are needed for the pipeline instead of `requirements.txt`. [Follow the instructions on how to run pipeline with Custom Container images](https://beam.apache.org/documentation/runtime/environments/#running-pipelines).
+
+1. If you are passing a custom container image, `--sdk_container_image` at runtime and specify `--requirements_file` option, we recommend you to install the dependencies from the `--requirements_file` when building your container image. In this case, you would reduce the pipeline startup time and do not need to pass `--requirements_file` option at runtime.


If you are using a custom container image, we recommend that you install the dependencies from the --requirements_file directly into your image at build time. In this case, you do not need to pass --requirements_file option at runtime, which will reduce the pipeline startup time. Fore example:...

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

tvalentyn · 2022-03-04T21:01:35Z

website/www/site/content/en/documentation/runtime/environments.md

@@ -171,6 +171,49 @@ creates a Java 8 SDK image with appropriate licenses in `/opt/apache/beam/third_

 By default, no licenses/notices are added to the docker images.

+#### Build an existing container image to make it compatible with Apache Beam Runners {#modify-existing-base-image}


@emilymye could you PTAL at this section?

Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>

y1chi · 2022-03-07T17:55:26Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+
+In the pre-building step, we install pipeline dependencies on the container image prior to the job submission. This would speed up the pipeline execution.\
+To use pre-building the dependencies from `requirements.txt` on the container image. Follow the steps below.
+1. Provide the container engine. We support `docker` and `cloud_build`(requires a GCP project with Cloud Build API enabled).


local_docker?

y1chi · 2022-03-07T17:57:11Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+1. Provide the container engine. We support `docker` and `cloud_build`(requires a GCP project with Cloud Build API enabled).
+
+       --prebuild_sdk_container_enginer <execution_environment>
+2. To pass a base image for pre-building dependencies, enable this flag. If not, apache beam's base image would be used.


Note that this may not work on an arbitrary base image, the base image should follow the same contract to install dependencies in a setup_only mode as apache beam's base image https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L49

prebuild_sdk_container_engine (not enginer)

Good point that the container needs to have the official entry point for this to work. I think in all container-customization mechanisms we suggest, one way or another we recommend, to use Beam's boot entry point.

As a part of making prebuilding not experimental, I think we should remove prebuild_sdk_container_base_image and just use --sdk_container_image flag for this purpose. i don't see the need for two different flags.

+1 to the point 3

@y1chi took care of #3 in #17032. Thanks, @y1chi .

As @y1chi pointed out, it may not work if the user doesn't follow apache beam's contract. But we do instruct them to follow the contract in some way.

So, I assume we can introduce this section as part of the instruction?

I would remove #2 now that we don't need a special flag and use the standard --sdk_container_image flag for this purpose.

I feel like we can keep 2 and update the pipeline option to --sdk_container_image=....

y1chi · 2022-03-07T17:57:27Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+2. To pass a base image for pre-building dependencies, enable this flag. If not, apache beam's base image would be used.
+
+       --prebuild_sdk_container_base_image <location_to_base_image>
+3. To push the container image, pre-built locally with `Docker` , to a remote repository(eg: docker registry), provide URL to the docker registry by passing


local_docker

Yes it should be local_docker

emilymye · 2022-03-07T18:36:55Z

website/www/site/content/en/documentation/runtime/environments.md

@@ -171,6 +171,49 @@ creates a Java 8 SDK image with appropriate licenses in `/opt/apache/beam/third_

 By default, no licenses/notices are added to the docker images.

+#### Build an existing container image to make it compatible with Apache Beam Runners {#modify-existing-base-image}


Suggested change

#### Build an existing container image to make it compatible with Apache Beam Runners {#modify-existing-base-image}

#### Build a container image based on an existing image compatible with Apache Beam Runners {#modify-existing-base-image}

emilymye · 2022-03-07T18:42:46Z

website/www/site/content/en/documentation/runtime/environments.md

@@ -171,6 +171,49 @@ creates a Java 8 SDK image with appropriate licenses in `/opt/apache/beam/third_

 By default, no licenses/notices are added to the docker images.

+#### Build an existing container image to make it compatible with Apache Beam Runners {#modify-existing-base-image}
+Beam offers a way to take a Beam container image and customize it. But if you have an existing base image to be compatible with Apache Beam Runners, use a [multi-stage build](https://docs.docker.com/develop/develop-images/multistage-build/) process to copy over the necessary artifacts from a default Apache Beam base image and provide your custom container image.


Suggested change

Beam offers a way to take a Beam container image and customize it. But if you have an existing base image to be compatible with Apache Beam Runners, use a [multi-stage build](https://docs.docker.com/develop/develop-images/multistage-build/) process to copy over the necessary artifacts from a default Apache Beam base image and provide your custom container image.

Beam offers a way to provide your own custom Beam container image. The easiest way to build a new custom image that is compatible with Apache Beam Runners is to use a [multi-stage build](https://docs.docker.com/develop/develop-images/multistage-build/) process. This copies over the necessary artifacts from a default Apache Beam base image to build your custom container image.

emilymye · 2022-03-07T18:43:16Z

website/www/site/content/en/documentation/runtime/environments.md

+ # Copy files from official SDK image, including script/dependencies.
+ COPY --from=apache/beam_python3.7_sdk:2.25.0 /opt/apache/beam /opt/apache/beam
+
+ # Perform any addtional customizations if desired


Suggested change

# Perform any addtional customizations if desired

# Perform any additional customizations if desired

emilymye · 2022-03-07T18:46:11Z

website/www/site/content/en/documentation/runtime/environments.md

+ FROM python:3.8-slim
+
+ # Install SDK. (needed for Python SDK)
+ RUN pip install --no-cache-dir apache-beam[gcp]==2.25.0


Can make the version here a build arg https://docs.docker.com/engine/reference/builder/#arg

e.g.

FROM python:3.8-slim ARG beam_ver=2.25.0 ... RUN pip install --no-cache-dir apache-beam[gcp]==$beam_ver COPY --from=apache/beam_python3.7_sdk:$beam_ver /opt/apache/beam /opt/apache/beam

I think we tried this before and it didn't work.

I built a Docker image with ARG BEAM_VERSION=2.31.0 and RUN pip install --no-cache-dir apache-beam[gcp]==$BEAM_VERSION. This seems to be working. I verified the /opt/apache/beam/boot path.

Can you recall what was the error?

I thought using Dockerfile params (ARG) in the FROM statement wasn't possible. That's what I was referring to.

Yes, expansion of ARG in the COPY --from is not supported in Docker. Hence moving forward with the current instructions

AnandInguva · 2022-03-09T17:58:12Z

PTAL @tvalentyn @emilymye @y1chi

y1chi · 2022-03-09T23:26:02Z

website/www/site/content/en/documentation/runtime/environments.md

@@ -171,6 +171,48 @@ creates a Java 8 SDK image with appropriate licenses in `/opt/apache/beam/third_

 By default, no licenses/notices are added to the docker images.

+#### Modifying an existing container image to make it compatible with Apache Beam Runners {#modify-existing-base-image}
+Beam offers a way to provide your own custom container image. The easiest way to build a new custom image that is compatible with Apache Beam Runners is to use a [multi-stage build](https://docs.docker.com/develop/develop-images/multistage-build/) process. This copies over the necessary artifacts from a default Apache Beam base image to build your custom container image.


Is this multi-stage build?

Yes. Using multi stage build process to copy required artifacts from Apache Beam's base image to the provided custom image

y1chi · 2022-03-09T23:41:39Z

website/www/site/content/en/documentation/runtime/environments.md

+ RUN pip install --no-cache-dir apache-beam[gcp]==2.35.0
+
+ # Copy files from official SDK image, including script/dependencies.
+ COPY --from=apache/beam_python3.7_sdk:2.35.0 /opt/apache/beam /opt/apache/beam


I wonder if this is sufficient, IIRC /opt/apache/beam this only contains the boot program? all the base_image_requirements are in site_packages?

they are in dist_packages

the requirements will be installed at runtime , when we pip-install the staged apache beam sdk here

beam/sdks/python/container/piputil.go

Line 164 in d46bd07

err := pipInstallPackage(files, workDir, sdkWhlFile, false, false, []string{"gcp"})

website/www/site/content/en/documentation/runtime/environments.md

codecov · 2022-03-17T00:52:59Z

Codecov Report

Merging #16938 (d46bd07) into master (2a45a5b) will increase coverage by 0.94%.
The diff coverage is 26.35%.

❗ Current head d46bd07 differs from pull request most recent head 9ad0ba9. Consider uploading reports for the commit 9ad0ba9 to get more accurate results

@@            Coverage Diff             @@
##           master   #16938      +/-   ##
==========================================
+ Coverage   73.00%   73.95%   +0.94%     
==========================================
  Files         658      669      +11     
  Lines       86706    88159    +1453     
==========================================
+ Hits        63301    65194    +1893     
+ Misses      22405    21853     -552     
- Partials     1000     1112     +112

Flag	Coverage Δ
go	`49.35% <26.35%> (+3.69%)`	⬆️
python	`83.64% <ø> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...s/go/pkg/beam/core/graph/window/trigger/trigger.go	`59.09% <0.00%> (ø)`
sdks/go/pkg/beam/core/graph/window/windows.go	`82.60% <ø> (ø)`
sdks/go/pkg/beam/core/runtime/exec/util.go	`74.28% <ø> (ø)`
sdks/go/pkg/beam/core/runtime/harness/harness.go	`11.26% <0.00%> (-1.43%)`	⬇️
sdks/go/pkg/beam/core/runtime/exec/plan.go	`48.61% <22.22%> (-8.80%)`	⬇️
sdks/go/pkg/beam/core/runtime/exec/pardo.go	`50.45% <25.00%> (ø)`
...kg/beam/core/runtime/xlangx/expansionx/download.go	`61.53% <61.53%> (+1.16%)`	⬆️
sdks/go/pkg/beam/core/runtime/exec/combine.go	`60.43% <66.66%> (ø)`
sdks/go/pkg/beam/core/runtime/exec/fn.go	`69.26% <80.00%> (+0.56%)`	⬆️
sdks/go/pkg/beam/core/runtime/harness/datamgr.go	`74.41% <100.00%> (ø)`
... and 60 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2a45a5b...9ad0ba9. Read the comment docs.

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

tvalentyn · 2022-03-17T06:37:49Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+
+## Pre-building SDK container image
+
+In the pre-building step, we install pipeline dependencies on the container image prior to the job submission. This would speed up the pipeline execution.\


Something is missing here. Let's add an introductory sentence.

In pipeline execution modes where a Beam runner launches SDK workers in Docker containers, the additional pipeline dependencies (specified via --requirements_file and other runtime options) are installed into the containers at runtime. This can increase the worker startup time. However, it may be possible to pre-build the SDK containers and perform the dependency installation once before the workers start.

tvalentyn · 2022-03-17T06:38:57Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+## Pre-building SDK container image
+
+In the pre-building step, we install pipeline dependencies on the container image prior to the job submission. This would speed up the pipeline execution.\
+To use pre-building the dependencies from `requirements.txt` on the container image. Follow the steps below.


Suggested change

To use pre-building the dependencies from `requirements.txt` on the container image. Follow the steps below.

To pre-build the container image before the pipeline submission, follow the steps below.

tvalentyn · 2022-03-17T06:43:50Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+To use pre-building the dependencies from `requirements.txt` on the container image. Follow the steps below.
+1. Provide the container engine. We support `local_docker` and `cloud_build`(requires a GCP project with Cloud Build API enabled).
+
+       --prebuild_sdk_container_engine <execution_environment>


Suggested change

--prebuild_sdk_container_engine <execution_environment>

--prebuild_sdk_container_engine <container_engine>

tvalentyn · 2022-03-17T06:44:13Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+
+In the pre-building step, we install pipeline dependencies on the container image prior to the job submission. This would speed up the pipeline execution.\
+To use pre-building the dependencies from `requirements.txt` on the container image. Follow the steps below.
+1. Provide the container engine. We support `local_docker` and `cloud_build`(requires a GCP project with Cloud Build API enabled).


Suggested change

1. Provide the container engine. We support `local_docker` and `cloud_build`(requires a GCP project with Cloud Build API enabled).

1. Provide the container engine. We support `local_docker` (requires local installation of Docker) and `cloud_build`(requires a GCP project with Cloud Build API enabled).

tvalentyn · 2022-03-17T06:46:51Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+2. To pass a base image for pre-building dependencies, enable this flag. If not, apache beam's base image would be used.
+
+       --sdk_container_image <location_to_base_image>
+3. To push the container image, pre-built locally with `local_docker` , to a remote repository(eg: docker registry), provide URL to the remote registry by passing


If using local_docker engine, provide a URL for the remote registry to which the image will be pushed by passing...

tvalentyn · 2022-03-17T06:48:33Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+       --sdk_container_image <location_to_base_image>
+3. To push the container image, pre-built locally with `local_docker` , to a remote repository(eg: docker registry), provide URL to the remote registry by passing
+
+       --docker_registry_push_url <IMAGE_URL>


I am confused - what is a sample value of this param? Is it supposed to be the image name+tag or just the registry?

It should be a registry. We generate the image tag. Image name is coded as beam_python_prebuilt_sdk at [1].

May be it can worded as --docker_registry_push_url <registry_URL>

[1]

beam/sdks/python/apache_beam/runners/portability/sdk_container_builder.py

Line 85 in 2a23166

container_image_name = os.path.join(

Can we give an example of the expected value? As a user reading this doc it is still not obvious.

We can add an example. Also let me see if I can make the wording more simpler

tvalentyn · 2022-03-17T06:48:57Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+1. Provide the container engine. We support `docker` and `cloud_build`(requires a GCP project with Cloud Build API enabled).
+
+       --prebuild_sdk_container_enginer <execution_environment>
+2. To pass a base image for pre-building dependencies, enable this flag. If not, apache beam's base image would be used.


I would remove #2 now that we don't need a special flag and use the standard --sdk_container_image flag for this purpose.

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

AnandInguva · 2022-03-18T19:38:39Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md


-       --docker_registry_push_url <IMAGE_URL>
+   **NOTE:** `docker_registry_push_url` must be a remote registry.


@y1chi if the user uses pre-building and doesn't provide docker_registry_push_url, what would happen in this case?

I recall it would fail with error something like this Couldn't find the Docker image. If this is the case, we need to make sure that user provides a remote registry URL. If the user doesn't provide it, can we fail the pipeline prior to Job submission?

AnandInguva · 2022-03-18T19:43:47Z

PTAL @tvalentyn

R: @pcoet @melap. Can you provide any suggestions/edits that can make the topic in PR more clear? Thanks!!! :)

pcoet

LGTM. Just a couple of minor suggestions.

pcoet · 2022-03-18T20:14:31Z

website/www/site/content/en/documentation/runtime/environments.md

+  ```
+>**NOTE**: This example assumes necessary dependencies (in this case, Python 3.7 and pip) have been installed on the existing base image. Installing the Apache Beam SDK into the image will ensure that the image has the necessary SDK dependencies and reduce the worker startup time.
+>The version specified in the `RUN` instruction must match the version used to launch the pipeline.<br>
+>**Users need to make sure that whatever base image they use has the same Python/Java interpreter version that they used to run the pipeline**.


Consider: "Make sure that the Python or Java runtime version specified in the base image is the same as the version used to run the pipeline."

pcoet · 2022-03-18T20:15:53Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

@@ -45,6 +45,17 @@ If your pipeline uses public packages from the [Python Package Index](https://py
    The runner will use the `requirements.txt` file to install your additional dependencies onto the remote workers.

 **Important:** Remote workers will install all packages listed in the `requirements.txt` file. Because of this, it's very important that you delete non-PyPI packages from the `requirements.txt` file, as stated in step 2. If you don't remove non-PyPI packages, the remote workers will fail when attempting to install packages from sources that are unknown to them.
+> **NOTE**: An alternative to `pip freeze` is to use a library like [pip-tools](https://github.com/jazzband/pip-tools) to compile the all the dependencies required for the pipeline from a `--requirements_file`, where only top-level dependencies are mentioned.


"compile the all" -> "compile all"

pcoet · 2022-03-18T20:18:06Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+
+In pipeline execution modes where a Beam runner launches SDK workers in Docker containers, the additional pipeline dependencies (specified via `--requirements_file` and other runtime options) are installed into the containers at runtime. This can increase the worker startup time.
+ However, it may be possible to pre-build the SDK containers and perform the dependency installation once before the workers start. To pre-build the container image before pipeline submission, provide the pipeline options mentioned below.
+1. Provide the container engine. We support `local_docker`(requires local installation of Docker) and `cloud_build`(requires a GCP project with Cloud Build API enabled).


In general, prefer "Beam" to "we", as in "Beam supports..."

tvalentyn · 2022-03-22T16:52:45Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+1. Provide the container engine. Beam supports `local_docker`(requires local installation of Docker) and `cloud_build`(requires a GCP project with Cloud Build API enabled).
+
+       --prebuild_sdk_container_engine=<container_engine>
+2. To pass a base image for pre-building dependencies, provide `--sdk_container_image`. If not, Apache beam's base [image](https://hub.docker.com/search?q=apache%2Fbeam&type=image) would be used.


As discussed offline, let's remove this and line 156.

tvalentyn · 2022-03-22T17:12:20Z

website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md

+
+   **NOTE:** `docker_registry_push_url` must be a remote registry.
+> To use Docker, the `--sdk_container_image` should be compatible with Apache Beam Runner. Please follow the [instructions](https://beam.apache.org/documentation/runtime/environments/#building-and-pushing-custom-containers) on how to build a base container image compatible with Apache Beam.
+


Suggestion to add to the notes:

The pre-building feature requires the Apache Beam SDK for Python, version 2.25.0 or later.

The container images created during prebuilding will persist beyond the pipeline runtime.
Once your job is finished or stopped, you can remove the pre-built image from the container registry.

If your pipeline is using a custom container image, most likely you will not benefit from prebuilding step as extra dependencies can be preinstalled in the custom image at build time. If you still would like to use prebuilding with custom images, use Apache Beam SDK 2.38.0 or newer and supply your custom image in via the --sdk_container_image pipeline option.

tvalentyn · 2022-03-29T15:35:02Z

Thanks!

AnandInguva added 3 commits February 24, 2022 10:16

Added custom containers

d00f744

Add instructions for pre-building container image

68e851f

Add documentation on using custom containers, prebuilding workflows

072ead7

github-actions bot added the website label Feb 24, 2022

AnandInguva force-pushed the BEAM-13314 branch from f504f4f to ac660b2 Compare February 24, 2022 19:52

Fixup: whitespaces

422d9ba

AnandInguva force-pushed the BEAM-13314 branch from ac660b2 to 422d9ba Compare February 24, 2022 20:00

AnandInguva changed the title ~~[Beam 13314]Revise recommendations to manage Python pipeline dependencies.~~ [BEAM-13314]Revise recommendations to manage Python pipeline dependencies. Feb 25, 2022

tvalentyn reviewed Mar 4, 2022

View reviewed changes

AnandInguva and others added 2 commits March 7, 2022 11:58

Fix documentation

f9ec6f4

Apply suggestions from code review

98facb0

Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>

y1chi reviewed Mar 7, 2022

View reviewed changes

emilymye reviewed Mar 7, 2022

View reviewed changes

AnandInguva added 2 commits March 7, 2022 16:32

Fix typos and update documentation

d2b55ba

Change container image option for prebuilding

c367ab3

y1chi reviewed Mar 9, 2022

View reviewed changes

tvalentyn reviewed Mar 16, 2022

View reviewed changes

website/www/site/content/en/documentation/runtime/environments.md Show resolved Hide resolved

tvalentyn reviewed Mar 17, 2022

View reviewed changes

AnandInguva added 2 commits March 18, 2022 13:30

update documentation

f712244

Fixup: whitespace

758ee0a

AnandInguva commented Mar 18, 2022

View reviewed changes

pcoet approved these changes Mar 18, 2022

View reviewed changes

Fix up documentation

860c4f0

tvalentyn approved these changes Mar 22, 2022

View reviewed changes

Update doc

9ad0ba9

tvalentyn merged commit c84818d into apache:master Mar 29, 2022

	3. [Build](#modify-existing-base-image) an existing container image to make it compatible with Apache Beam Runners. This method is used when users start from an existing image, and configure the image to be compatible with Apache Beam Runners.
	3. [Modifying](#modify-existing-base-image) an existing container image to make it compatible with Apache Beam Runners. This method is used when users start from an existing image, and configure the image to be compatible with Apache Beam Runners.


		You can pass a [container](https://hub.docker.com/search?q=apache%2Fbeam&type=image) image with all the dependencies that are needed for the pipeline instead of `requirements.txt`. [Follow the instructions on how to run pipeline with Custom Container images](https://beam.apache.org/documentation/runtime/environments/#running-pipelines).

		1. If you are passing a custom container image, `--sdk_container_image` at runtime and specify `--requirements_file` option, we recommend you to install the dependencies from the `--requirements_file` when building your container image. In this case, you would reduce the pipeline startup time and do not need to pass `--requirements_file` option at runtime.

		@@ -171,6 +171,49 @@ creates a Java 8 SDK image with appropriate licenses in `/opt/apache/beam/third_

		By default, no licenses/notices are added to the docker images.

		#### Build an existing container image to make it compatible with Apache Beam Runners {#modify-existing-base-image}

	#### Build an existing container image to make it compatible with Apache Beam Runners {#modify-existing-base-image}
	#### Build a container image based on an existing image compatible with Apache Beam Runners {#modify-existing-base-image}

	Beam offers a way to take a Beam container image and customize it. But if you have an existing base image to be compatible with Apache Beam Runners, use a [multi-stage build](https://docs.docker.com/develop/develop-images/multistage-build/) process to copy over the necessary artifacts from a default Apache Beam base image and provide your custom container image.
	Beam offers a way to provide your own custom Beam container image. The easiest way to build a new custom image that is compatible with Apache Beam Runners is to use a [multi-stage build](https://docs.docker.com/develop/develop-images/multistage-build/) process. This copies over the necessary artifacts from a default Apache Beam base image to build your custom container image.

	# Perform any addtional customizations if desired
	# Perform any additional customizations if desired


		## Pre-building SDK container image

		In the pre-building step, we install pipeline dependencies on the container image prior to the job submission. This would speed up the pipeline execution.\

	To use pre-building the dependencies from `requirements.txt` on the container image. Follow the steps below.
	To pre-build the container image before the pipeline submission, follow the steps below.

	--prebuild_sdk_container_engine <execution_environment>
	--prebuild_sdk_container_engine <container_engine>

	1. Provide the container engine. We support `local_docker` and `cloud_build`(requires a GCP project with Cloud Build API enabled).
	1. Provide the container engine. We support `local_docker` (requires local installation of Docker) and `cloud_build`(requires a GCP project with Cloud Build API enabled).


		--docker_registry_push_url <IMAGE_URL>
		NOTE: `docker_registry_push_url` must be a remote registry.


		NOTE: `docker_registry_push_url` must be a remote registry.
		> To use Docker, the `--sdk_container_image` should be compatible with Apache Beam Runner. Please follow the [instructions](https://beam.apache.org/documentation/runtime/environments/#building-and-pushing-custom-containers) on how to build a base container image compatible with Apache Beam.

[BEAM-13314]Revise recommendations to manage Python pipeline dependencies. #16938

[BEAM-13314]Revise recommendations to manage Python pipeline dependencies. #16938

Conversation

AnandInguva commented Feb 24, 2022

GitHub Actions Tests Status (on master branch)

AnandInguva commented Feb 24, 2022

tvalentyn Mar 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tvalentyn Mar 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tvalentyn Mar 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnandInguva Mar 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnandInguva Mar 9, 2022 • edited Loading

Choose a reason for hiding this comment

AnandInguva commented Mar 9, 2022

Choose a reason for hiding this comment

AnandInguva Mar 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Mar 17, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnandInguva Mar 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tvalentyn Mar 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnandInguva commented Mar 18, 2022

pcoet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tvalentyn commented Mar 29, 2022

tvalentyn Mar 4, 2022 •

edited

Loading

tvalentyn Mar 7, 2022 •

edited

Loading

tvalentyn Mar 17, 2022 •

edited

Loading

AnandInguva Mar 7, 2022 •

edited

Loading

AnandInguva Mar 9, 2022 •

edited

Loading

AnandInguva Mar 10, 2022 •

edited

Loading

codecov bot commented Mar 17, 2022 •

edited

Loading

AnandInguva Mar 17, 2022 •

edited

Loading

tvalentyn Mar 17, 2022 •

edited

Loading