bootstrap: vendor crates required by opt-dist to collect profiles #125465

weihanglo · 2024-05-23T22:14:32Z

These are the default package set required by opt-dist to correctly work,
hence for people wanting to build a production grade of rustc in a
sandboxed / air-gapped environment, these need to be vendored.

The size of rustc-src-nightly.tar.xz before and after this change:

Before: 298M
After: 323M (+8%)

Size change might or might not be a concern.
See the previous discussion: #125166 (comment)

Previous efforts on making:

Note that extra works still need to be done to make it fully vendored.

The current pinned rustc-perf uses tempfile::Tempdir as the working
directory when collecting profiles from some of these packages.
This "tmp" working directory usage make it impossible for Cargo to pick
up the correct vendor sources setting in .cargo/config.toml bundled
in the rustc-src tarball. ¹
opt-dist verifies the final built rustc against a subset of rustc test
suite. However it rolls out its own config.toml without setting
vendor = true, and that results in ./vendor/ directory removed.
²

rustbot · 2024-05-23T22:14:41Z

r? @albertlarsan68

rustbot has assigned @albertlarsan68.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

weihanglo · 2024-05-23T22:23:41Z

r? @Mark-Simulacrum as you reviewed the previous two PRs.
cc @Kobzol as you have been working hard in this area :)

Kobzol · 2024-05-24T13:20:21Z

r? @Kobzol

This makes sense, and it is nicer than packaging everything from rustc-perf, which is quite large.

@bors r+ rollup

bors · 2024-05-24T13:20:24Z

📌 Commit 51a87fc has been approved by Kobzol

It is now in the queue for this repository.

bootstrap: vendor crates required by opt-dist to collect profiles These are the default package set required by opt-dist to correctly work, hence for people wanting to build a production grade of rustc in a sandboxed / air-gapped environment, these need to be vendored. The size of `rustc-src-nightly.tar.xz` before and after this change: * Before: 298M * After: 323M (+8%) Size change might or might not be a concern. See the previous discussion: rust-lang#125166 (comment) Previous efforts on making: * rust-lang#125125 * rust-lang#125166 --- Note that extra works still need to be done to make it fully vendored. * The current pinned rustc-perf uses `tempfile::Tempdir` as the working directory when collecting profiles from some of these packages. This "tmp" working directory usage make it impossible for Cargo to pick up the correct vendor sources setting in `.cargo/config.toml` bundled in the rustc-src tarball. [^1] * opt-dist verifies the final built rustc against a subset of rustc test suite. However it rolls out its own `config.toml` without setting `vendor = true`, and that results in `./vendor/` directory removed. [^2] [^1]: https://github.com/rust-lang/rustc-perf/blob/4f313add609f43e928e98132358e8426ed3969ae/collector/src/compile/benchmark/mod.rs#L164-L173 [^2]: https://github.com/rust-lang/rust/blob/606afbb617a2949a4e35c4b0258ff94c980b9451/src/tools/opt-dist/src/tests.rs#L62-L77

…ml, r=<try> fix(opt-dist): respect existing config.toml This is another step toward making opt-dist work in sandboxed environments. See also <rust-lang#125465>. opt-dist verifies the final built rustc against a subset of rustc test suite. However it overwrote the pre-existing `config.toml` [^1], and that results in ./vendor/ directory removed [^2]. Instead of overwriting, this patch use `--set <config-value>` to override paths to rustc / cargo / llvm-config. [^1]: https://github.com/rust-lang/rust/blob/606afbb617a2949a4e35c4b0258ff94c980b9451/src/tools/opt-dist/src/tests.rs#L62-L77 [^2]: https://github.com/rust-lang/rust/blob/8679004993f08807289911d9f400f4ac4391d2bc/src/bootstrap/bootstrap.py#L1057

…iaskrgr Rollup of 6 pull requests Successful merges: - rust-lang#125455 (Make `clamp` inline) - rust-lang#125465 (bootstrap: vendor crates required by opt-dist to collect profiles ) - rust-lang#125477 (Run rustfmt on files that need it.) - rust-lang#125481 (Fix the dead link in the bootstrap README) - rust-lang#125482 (Notify kobzol after changes to `opt-dist`) - rust-lang#125489 (Revert problematic opaque type change) r? `@ghost` `@rustbot` modify labels: rollup

matthiaskrgr · 2024-05-24T15:47:30Z

@bors r-
I guess this failed here #125491 (comment)

Kobzol · 2024-05-24T18:53:07Z

That seems unlikely, that failure looks like a spurious network error. It happened even before bootstrap was invoked, and in a builder that doesn't even do dist.

Kobzol · 2024-05-24T18:55:20Z

@bors r+

bors · 2024-05-24T18:55:22Z

📌 Commit 51a87fc has been approved by Kobzol

It is now in the queue for this repository.

bjorn3 · 2024-05-24T19:14:23Z

Maybe add them to tidy's license check too? Without this PR they can't be added as the license check needs deps to be either vendored or internet access to be allowed and internet access is not allowed in all contexts where tidy runs. And it makes sense to me to run the license checker on everything we ship in either binary or source form to avoid surprises.

Kobzol · 2024-05-24T19:26:45Z

Does tidy check the licenses of all the dependencies and crates that we vendor? E.g. all dependencies of cargo etc.?

weihanglo · 2024-05-24T20:08:18Z

For cargo part, yes. I don't know about other things.

Kobzol · 2024-06-05T11:56:46Z

I'm not sure about the licensing part, but the rest looks good to me. Have you tried to do a vendored build of opt-dist, to see if it works end-to-end? We probably won't have any testing for this.

(Big thanks for pushing all these changes through, btw! It's great to provide the optimized workflow for more people that build their own compiler).

weihanglo · 2024-06-07T05:53:52Z

@Kobzol yes. I ran src/ci/docker/run.sh dist-x86_64-linux with some modifications and it works. No crate was downloaded during the build. Verified by checking the existence of $CARGO_HOME/registry,git}. Without --benchmark-cargo-config provided crates will be downloaded.

Click to see the diff (basically call opt-dist local with linux-ci args)

diff --git a/src/ci/docker/host-x86_64/dist-x86_64-linux/Dockerfile b/src/ci/docker/host-x86_64/dist-x86_64-linux/Dockerfile
index 4e1aae98221..f1911fdea96 100644
--- a/src/ci/docker/host-x86_64/dist-x86_64-linux/Dockerfile
+++ b/src/ci/docker/host-x86_64/dist-x86_64-linux/Dockerfile
@@ -66,6 +66,7 @@ ENV PGO_HOST=x86_64-unknown-linux-gnu
 ENV HOSTS=x86_64-unknown-linux-gnu
 
 ENV RUST_CONFIGURE_ARGS \
+      --enable-vendor \
       --enable-full-tools \
       --enable-sanitizers \
       --enable-profiler \
@@ -80,10 +81,22 @@ ENV RUST_CONFIGURE_ARGS \
       --set rust.lto=thin \
       --set rust.codegen-units=1
 
-ENV SCRIPT python3 ../x.py build --set rust.debug=true opt-dist && \
-    ./build/$HOSTS/stage0-tools-bin/opt-dist linux-ci -- python3 ../x.py dist \
+ENV SCRIPT mkdir -p /tmp/tmp-multistage/ && \
+  ln -s /checkout/.cargo /tmp/tmp-multistage/.cargo && \
+  ln -s /checkout/vendor /tmp/tmp-multistage/vendor && \
+  python3 ../x.py build --set rust.debug=true opt-dist && \
+    ./build/$HOSTS/stage0-tools-bin/opt-dist local \
+    --target-triple x86_64-unknown-linux-gnu \
+    --checkout-dir /checkout \
+    --artifact-dir /tmp/tmp-multistage/opt-artifacts \
+    --llvm-dir /rustroot \
+    --llvm-shared \
+    --benchmark-cargo-config /checkout/.cargo/config.toml \
+    --use-bolt \
+    --skipped-tests tests/ui/process/nofile-limit.rs \
+    -- python3 ../x.py dist \
     --host $HOSTS --target $HOSTS \
     --include-default-paths \
     build-manifest bootstrap
 ENV CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=clang
 
diff --git a/src/tools/opt-dist/src/main.rs b/src/tools/opt-dist/src/main.rs
index e4271a6e2dd..c141ff72c9a 100644
--- a/src/tools/opt-dist/src/main.rs
+++ b/src/tools/opt-dist/src/main.rs
@@ -132,7 +132,7 @@ fn create_environment(args: Args) -> anyhow::Result<(Environment, Vec<String>)>
                 .checkout_dir(checkout_dir.clone())
                 .host_llvm_dir(llvm_dir)
                 .artifact_dir(artifact_dir)
-                .build_dir(checkout_dir)
+                .build_dir(checkout_dir.join("obj"))
                 .prebuilt_rustc_perf(rustc_perf_checkout_dir)
                 .shared_llvm(llvm_shared)
                 .use_bolt(use_bolt)

Kobzol · 2024-06-07T07:08:12Z

Awesome :)

weihanglo · 2024-06-07T16:03:14Z

rust-lang/team and rust-lang/measureme have no license. I guess it's fine just adding one for them?

Yeah, we should propagate upstream and get approvals from relevant contributors@m for each of those repos (shouldn't be many people).

@Mark-Simulacrum I think the license and git source issues have been resolved. Would you mind reviewing this? We're vendoring rustc-perf in 1.80.0, so it might better for this to go hand-in-hand with that before new nightly branches off.

Mark-Simulacrum · 2024-06-07T16:18:26Z

Nightly probably already branched, but I'll take a look on the weekend and maybe nominate for beta backport depending on details.

src/tools/build_helper/src/metrics.rs

src/tools/tidy/src/deps.rs

Mark-Simulacrum · 2024-06-09T16:25:49Z

r=me with nits fixed

maybe nominate for beta backport depending on details.

I think given the changes here I'm not currently inclined to backport it.

These are the default package set required by opt-dist to correctly work, hence for people wanting to build a production grade of rustc in a sandboxed / air-gapped environment, these need to be vendored. The size of `rustc-src-nightly.tar.xz` before and after this change: * Before: 298M * After: 323M (+8%) These crates are the default set of packages required by opt-dist to correctly work, hence for people wanting to build a production grade of rustc in an sandboxed / air-gapped environment, these need to be vendored. The size of `rustc-src-nightly.tar.xz` before and after this change: * Before: 298M * After: 323M (+8%) Size change might or might not be a concern. See the previous discussion: rust-lang#125166 (comment) Previous efforts on making: * rust-lang#125125 * rust-lang#125166 --- Note that extra works still need to be done to make it fully vendored. * The current pinned rustc-perf uses `tempfile::Tempdir` as the working directory when collecting profiles from some of these packages. This "tmp" working directory usage make it impossible for Cargo to pick up the correct vendor sources setting in `.cargo/config.toml` bundled in the rustc-src tarball. [^1] * opt-dist verifies the final built rustc against a subset of rustc test suite. However it rolls out its own `config.toml` without setting `vendor = true`, and that results in `./vendor/` directory removed. [^2] [^1]: https://github.com/rust-lang/rustc-perf/blob/4f313add609f43e928e98132358e8426ed3969ae/collector/src/compile/benchmark/mod.rs#L164-L173 [^2]: https://github.com/rust-lang/rust/blob/606afbb617a2949a4e35c4b0258ff94c980b9451/src/tools/opt-dist/src/tests.rs#L62-L77

`ring` is included because it is an optional dependency of `hyper`, which is a training data in rustc-pref for optimized build. The license of it is generally `ISC AND MIT AND OpenSSL`, though the `package.license` field is not set. See briansmith/ring#902 `git+https://github.com/rust-lang/team` git source is from `rust_team_data`, which is used by `site` in src/tools/rustc-perf. This doesn't really a part of the compiler, so doesn't really affect the bootstrapping process. See rust-lang/rustc-perf#1914.

weihanglo · 2024-06-09T18:03:05Z

nits are fixed, and the CI is all green now. Thanks for the review people.

@bors r=Mark-Simulacrum

bors · 2024-06-09T18:03:08Z

📌 Commit 3778703 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

…Simulacrum bootstrap: vendor crates required by opt-dist to collect profiles These are the default package set required by opt-dist to correctly work, hence for people wanting to build a production grade of rustc in a sandboxed / air-gapped environment, these need to be vendored. The size of `rustc-src-nightly.tar.xz` before and after this change: * Before: 298M * After: 323M (+8%) Size change might or might not be a concern. See the previous discussion: rust-lang#125166 (comment) Previous efforts on making: * rust-lang#125125 * rust-lang#125166 --- Note that extra works still need to be done to make it fully vendored. * The current pinned rustc-perf uses `tempfile::Tempdir` as the working directory when collecting profiles from some of these packages. This "tmp" working directory usage make it impossible for Cargo to pick up the correct vendor sources setting in `.cargo/config.toml` bundled in the rustc-src tarball. [^1] * opt-dist verifies the final built rustc against a subset of rustc test suite. However it rolls out its own `config.toml` without setting `vendor = true`, and that results in `./vendor/` directory removed. [^2] [^1]: https://github.com/rust-lang/rustc-perf/blob/4f313add609f43e928e98132358e8426ed3969ae/collector/src/compile/benchmark/mod.rs#L164-L173 [^2]: https://github.com/rust-lang/rust/blob/606afbb617a2949a4e35c4b0258ff94c980b9451/src/tools/opt-dist/src/tests.rs#L62-L77

Rollup of 6 pull requests Successful merges: - rust-lang#125041 (Enable GVN for `AggregateKind::RawPtr`) - rust-lang#125253 (Add `FRAC_1_SQRT_2PI` constant to f16/f32/f64/f128) - rust-lang#125465 (bootstrap: vendor crates required by opt-dist to collect profiles ) - rust-lang#125470 (Add release notes for 1.79.0) - rust-lang#125963 (Remove hard-coded hashes from codegen tests) - rust-lang#126188 (Fix documentation for `impl_common_helpers` in `run-make-support`) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#125465 - weihanglo:opt-dist-vendor, r=Mark-Simulacrum bootstrap: vendor crates required by opt-dist to collect profiles These are the default package set required by opt-dist to correctly work, hence for people wanting to build a production grade of rustc in a sandboxed / air-gapped environment, these need to be vendored. The size of `rustc-src-nightly.tar.xz` before and after this change: * Before: 298M * After: 323M (+8%) Size change might or might not be a concern. See the previous discussion: rust-lang#125166 (comment) Previous efforts on making: * rust-lang#125125 * rust-lang#125166 --- Note that extra works still need to be done to make it fully vendored. * The current pinned rustc-perf uses `tempfile::Tempdir` as the working directory when collecting profiles from some of these packages. This "tmp" working directory usage make it impossible for Cargo to pick up the correct vendor sources setting in `.cargo/config.toml` bundled in the rustc-src tarball. [^1] * opt-dist verifies the final built rustc against a subset of rustc test suite. However it rolls out its own `config.toml` without setting `vendor = true`, and that results in `./vendor/` directory removed. [^2] [^1]: https://github.com/rust-lang/rustc-perf/blob/4f313add609f43e928e98132358e8426ed3969ae/collector/src/compile/benchmark/mod.rs#L164-L173 [^2]: https://github.com/rust-lang/rust/blob/606afbb617a2949a4e35c4b0258ff94c980b9451/src/tools/opt-dist/src/tests.rs#L62-L77

michaelwoerister · 2024-06-10T07:41:27Z

🎉

…r-ozkan Update `rustc-perf` submodule before running tidy Since rust-lang#125465, `tidy` checks `src/tools/rustc-perf`, so we need to have it checked out before running `tidy`. Fixes: rust-lang#126224 r? `@onur-ozkan`

This is needed for fixing the missing license issue. See rust-lang#125465.

rustbot assigned albertlarsan68 May 23, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels May 23, 2024

rustbot assigned Mark-Simulacrum and unassigned albertlarsan68 May 23, 2024

weihanglo mentioned this pull request May 24, 2024

fix(opt-dist): respect existing config.toml #125473

Merged

rustbot assigned Kobzol and unassigned Mark-Simulacrum May 24, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 24, 2024

matthiaskrgr mentioned this pull request May 24, 2024

Rollup of 6 pull requests #125491

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels May 24, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 24, 2024

Mark-Simulacrum added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 7, 2024

Mark-Simulacrum self-assigned this Jun 7, 2024

Mark-Simulacrum reviewed Jun 9, 2024

View reviewed changes

src/tools/build_helper/src/metrics.rs Outdated Show resolved Hide resolved

src/tools/tidy/src/deps.rs Outdated Show resolved Hide resolved

weihanglo added 2 commits June 9, 2024 12:33

weihanglo force-pushed the opt-dist-vendor branch from 6799ee1 to 3778703 Compare June 9, 2024 17:05

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 9, 2024

This was referenced Jun 9, 2024

Rollup of 7 pull requests #126201

Closed

Rollup of 6 pull requests #126202

Merged

bors merged commit 4ca52f1 into rust-lang:master Jun 9, 2024
6 checks passed

rustbot added this to the 1.81.0 milestone Jun 9, 2024

weihanglo deleted the opt-dist-vendor branch June 9, 2024 21:16

Kobzol mentioned this pull request Jun 10, 2024

Update rustc-perf submodule before running tidy #126225

Merged

lcnr pushed a commit to lcnr/rust that referenced this pull request Jun 12, 2024

chore: update src/tools/rustc-perf

6d6c78e

This is needed for fixing the missing license issue. See rust-lang#125465.

weihanglo mentioned this pull request Sep 9, 2024

rust 1.81.0 fails to build without network access #130099

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bootstrap: vendor crates required by opt-dist to collect profiles #125465

bootstrap: vendor crates required by opt-dist to collect profiles #125465

weihanglo commented May 23, 2024 •

edited

Loading

rustbot commented May 23, 2024

weihanglo commented May 23, 2024

Kobzol commented May 24, 2024

bors commented May 24, 2024

matthiaskrgr commented May 24, 2024

Kobzol commented May 24, 2024 •

edited

Loading

Kobzol commented May 24, 2024

bors commented May 24, 2024

bjorn3 commented May 24, 2024 •

edited

Loading

Kobzol commented May 24, 2024

weihanglo commented May 24, 2024

Kobzol commented Jun 5, 2024

weihanglo commented Jun 7, 2024 •

edited

Loading

Kobzol commented Jun 7, 2024

weihanglo commented Jun 7, 2024

Mark-Simulacrum commented Jun 7, 2024

Mark-Simulacrum commented Jun 9, 2024

weihanglo commented Jun 9, 2024

bors commented Jun 9, 2024

michaelwoerister commented Jun 10, 2024

bootstrap: vendor crates required by opt-dist to collect profiles #125465

bootstrap: vendor crates required by opt-dist to collect profiles #125465

Conversation

weihanglo commented May 23, 2024 • edited Loading

Footnotes

rustbot commented May 23, 2024

weihanglo commented May 23, 2024

Kobzol commented May 24, 2024

bors commented May 24, 2024

matthiaskrgr commented May 24, 2024

Kobzol commented May 24, 2024 • edited Loading

Kobzol commented May 24, 2024

bors commented May 24, 2024

bjorn3 commented May 24, 2024 • edited Loading

Kobzol commented May 24, 2024

weihanglo commented May 24, 2024

Kobzol commented Jun 5, 2024

weihanglo commented Jun 7, 2024 • edited Loading

Kobzol commented Jun 7, 2024

weihanglo commented Jun 7, 2024

Mark-Simulacrum commented Jun 7, 2024

Mark-Simulacrum commented Jun 9, 2024

weihanglo commented Jun 9, 2024

bors commented Jun 9, 2024

michaelwoerister commented Jun 10, 2024

weihanglo commented May 23, 2024 •

edited

Loading

Kobzol commented May 24, 2024 •

edited

Loading

bjorn3 commented May 24, 2024 •

edited

Loading

weihanglo commented Jun 7, 2024 •

edited

Loading