rustbuild: Quickly dist cross-host compilers
#38731
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
r? @aturon (rust_highfive has picked a reviewer for you, use r? to override) |
|
r? @brson |
|
The first commit comes from #38667, so it can largely be ignored. |
brson
reviewed
Jan 3, 2017
| let hosts = if rule.only_host_build || rule.only_build { | ||
| &self.build.config.host[..1] | ||
| } else if self.build.flags.host.len() > 0 { | ||
| &self.build.flags.target |
There was a problem hiding this comment.
Should this be flags.host not flags.target?
|
r=me with comment addressed |
alexcrichton
force-pushed
the
supafast-cross-dist
branch
from
a2173f7
to
750aa85
Compare
|
@bors: r=brson |
|
📌 Commit 750aa85 has been approved by |
bors
added a commit
that referenced
this pull request
Jan 4, 2017
rustbuild: Quickly `dist` cross-host compilers
This commit optimizes the compile time for creating tarballs of cross-host
compilers and as a proof of concept adds two to the standard Travis matrix. Much
of this commit is further refactoring and refining of the `step.rs` definitions
along with the interpretation of `--target` and `--host` flags. This has gotten
confusing enough that I've also added a small test suite to
`src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress.
After this commit when you execute:
./x.py dist --host $MY_HOST --target $MY_HOST
the build system will compile two compilers. The first is for the build platform
and the second is for the host platform. This second compiler is then packaged
up and placed into `build/dist` and is ready to go. With a fully cached LLVM and
docker image I was able to create a cross-host compiler in around 20 minutes
locally.
Eventually we plan to add a whole litany of cross-host entries to the Travis
matrix, but for now we're just adding a few before we eat up all the extra
capacity.
cc #38531
|
💔 Test failed - status-travis |
This commit optimizes the compile time for creating tarballs of cross-host
compilers and as a proof of concept adds two to the standard Travis matrix. Much
of this commit is further refactoring and refining of the `step.rs` definitions
along with the interpretation of `--target` and `--host` flags. This has gotten
confusing enough that I've also added a small test suite to
`src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress.
After this commit when you execute:
./x.py dist --host $MY_HOST --target $MY_HOST
the build system will compile two compilers. The first is for the build platform
and the second is for the host platform. This second compiler is then packaged
up and placed into `build/dist` and is ready to go. With a fully cached LLVM and
docker image I was able to create a cross-host compiler in around 20 minutes
locally.
Eventually we plan to add a whole litany of cross-host entries to the Travis
matrix, but for now we're just adding a few before we eat up all the extra
capacity.
cc rust-lang#38531
alexcrichton
force-pushed
the
supafast-cross-dist
branch
from
750aa85
to
1a040b3
Compare
|
📌 Commit 1a040b3 has been approved by |
|
Ahhh indeed... I was only looking at the logic changes and overlooked that part of diff. XD |
|
⌛ Testing commit 1a040b3 with merge 80745e2... |
bors
added a commit
that referenced
this pull request
Jan 5, 2017
rustbuild: Quickly `dist` cross-host compilers
This commit optimizes the compile time for creating tarballs of cross-host
compilers and as a proof of concept adds two to the standard Travis matrix. Much
of this commit is further refactoring and refining of the `step.rs` definitions
along with the interpretation of `--target` and `--host` flags. This has gotten
confusing enough that I've also added a small test suite to
`src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress.
After this commit when you execute:
./x.py dist --host $MY_HOST --target $MY_HOST
the build system will compile two compilers. The first is for the build platform
and the second is for the host platform. This second compiler is then packaged
up and placed into `build/dist` and is ready to go. With a fully cached LLVM and
docker image I was able to create a cross-host compiler in around 20 minutes
locally.
Eventually we plan to add a whole litany of cross-host entries to the Travis
matrix, but for now we're just adding a few before we eat up all the extra
capacity.
cc #38531
|
☀️ Test successful - status-appveyor, status-travis |
alexcrichton
added a commit
to alexcrichton/rust
that referenced
this pull request
Jan 15, 2017
This commit adds six new travis matrix entires for doing cross-compiled distribution builds of the compiler. The support added in rust-lang#38731 allows us to quickly compile a complete suite of distribution artifacts for cross-compiled platforms, and currently each matrix entry (when fully cached) clocks in around an hour to finish. Note that a full test run typically takes about two hours right now. With further optimizations coming down the pike in rust-lang#39026 this commit also starts doubling up cross-compiled distribution builders on each matrix entry. We initially planned to do one build per entry, but it's looking like we may be able to get by with more than one in each entry. Depending on how long these builds take we may even be able to up it to three, but we'll start with two first. This commit then completes the suite of cross-compiled compilers that we're going to compile, adding it for a whole litany of platforms detailed in the changes to the docker files here. The existing `cross` image is also trimmed down quite a bit to avoid duplicate work, and we'll eventually provision it for far more cross compilation as well. Note that the gcc toolchains installed to compile most of these compilers are inappropriate for actualy distribution. The glibc they pull in is much newer than we'd like, so before we turn nightlies off we'll need to tweak these docker files to custom build toolchains like the current `linux-cross` docker image does.
bors
added a commit
that referenced
this pull request
Jan 15, 2017
travis: Expand dist builder coverage This commit adds six new travis matrix entires for doing cross-compiled distribution builds of the compiler. The support added in #38731 allows us to quickly compile a complete suite of distribution artifacts for cross-compiled platforms, and currently each matrix entry (when fully cached) clocks in around an hour to finish. Note that a full test run typically takes about two hours right now. With further optimizations coming down the pike in #39026 this commit also starts doubling up cross-compiled distribution builders on each matrix entry. We initially planned to do one build per entry, but it's looking like we may be able to get by with more than one in each entry. Depending on how long these builds take we may even be able to up it to three, but we'll start with two first. This commit then completes the suite of cross-compiled compilers that we're going to compile, adding it for a whole litany of platforms detailed in the changes to the docker files here. The existing `cross` image is also trimmed down quite a bit to avoid duplicate work, and we'll eventually provision it for far more cross compilation as well. Note that the gcc toolchains installed to compile most of these compilers are inappropriate for actualy distribution. The glibc they pull in is much newer than we'd like, so before we turn nightlies off we'll need to tweak these docker files to custom build toolchains like the current `linux-cross` docker image does.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit optimizes the compile time for creating tarballs of cross-host
compilers and as a proof of concept adds two to the standard Travis matrix. Much
of this commit is further refactoring and refining of the
step.rsdefinitionsalong with the interpretation of
--targetand--hostflags. This has gottenconfusing enough that I've also added a small test suite to
src/bootstrap/step.rsto ensure what we're doing works and doesn't regress.After this commit when you execute:
the build system will compile two compilers. The first is for the build platform
and the second is for the host platform. This second compiler is then packaged
up and placed into
build/distand is ready to go. With a fully cached LLVM anddocker image I was able to create a cross-host compiler in around 20 minutes
locally.
Eventually we plan to add a whole litany of cross-host entries to the Travis
matrix, but for now we're just adding a few before we eat up all the extra
capacity.
cc #38531