rustbuild: Add sccache support #38359
Conversation
|
r? @aturon (rust_highfive has picked a reviewer for you, use r? to override) |
alexcrichton
force-pushed
the
sccache
branch
from
f7e4ac3
to
9406a04
Compare
|
cc @luser, you may be interested here in the changes as well |
alexcrichton
changed the title
|
Looks like a fully cached build on Linux takes around 220 seconds with sccache and previously took about 70 seconds with ccache. The difference, 150 seconds, 2.5 minutes, seems negligible. |
|
Caches appear to not be working on OSX on travis. Sounds like a problem for Wednesday Alex. |
|
Oh right, secret vars aren't available on PRs, that explains the lack of caching. |
alexcrichton
force-pushed
the
sccache
branch
2 times, most recently
from
b50de70
to
4f60e74
Compare
After thinking about this some more, I suspect that sccache is not going to offer any benefit over your existing ccache setup with Travis on linux/osx. Given that you don't update LLVM at a rapid pace, and that Travis lets you persist the ccache cache between builders, you're probably already getting a completely cached build with every commit that's not an LLVM update. If the cache contents aren't changing very much then fetching the entire thing in one HTTP request is going to be faster than one HTTP request per object file. Our use case in Gecko is a bit different--we have a lot of C++ source files that change often, so sharing the entire ccache cache the way that Travis caches work would have huge overhead for us. @glandium's original comparisons were mostly about "using a local cccache cache" vs. "using sccache". All that being said, the majority of this patch will be useful regardless once we iron out the blockers to using sccache for your Windows builds, and it will also be useful if we get sccache working for Rust compilation. Also I wrote some inline comments on your changeset but I was not quite awake enough so I put them in the wrong place (oops): |
|
☔ The latest upstream changes (presumably #38340) made this pull request unmergeable. Please resolve the merge conflicts. |
Yes while it's true that we won't get speed wins I do think we'll get wins along lines of unified caching strategies across Linux/Mac/Windows plus a much smaller cache size. Right now the cache is duplicated across all Travis builders but they're all caching the same thing! Ideally this'd also extend to contributors one day as well, so I don't mind betting big on sccache in the meantime :) |
alexcrichton
force-pushed
the
sccache
branch
from
4f60e74
to
bacf18d
Compare
alexcrichton
changed the title
This commit adds support for sccache, a ccache-like compiler which works on MSVC and stores results into an S3 bucket. This also switches over all Travis and AppVeyor automation to using sccache to ensure a shared and unified cache over time which can be shared across builders. The support for sccache manifests as a new `--enable-sccache` option which instructs us to configure LLVM differently to use a 'sccache' binary instead of a 'ccache' binary. All docker images for Travis builds are updated to download Mozilla's tooltool builds of sccache onto various containers and systems. Additionally a new `rust-lang-ci-sccache` bucket is configured to hold all of our ccache goodies.
alexcrichton
force-pushed
the
sccache
branch
from
bacf18d
to
96a5fc7
Compare
|
Ok I believe this is working everywhere now, and I wouldn't mind starting to throw this at AppVeyor and Travis. I believe AppVeyor will continue to be cached via the existing caching mechanisms, and Travis will start caching through sccache instead. r? @brson, curious what you think as well for this. |
|
@bors r+ |
|
📌 Commit 96a5fc7 has been approved by |
|
⌛ Testing commit 96a5fc7 with merge d250169... |
rustbuild: Add sccache support This commit adds support for sccache, a ccache-like compiler which works on MSVC and stores results into an S3 bucket. This also switches over all Travis and AppVeyor automation to using sccache to ensure a shared and unified cache over time which can be shared across builders. The support for sccache manifests as a new `--enable-sccache` option which instructs us to configure LLVM differently to use a 'sccache' binary instead of a 'ccache' binary. All docker images for Travis builds are updated to download Mozilla's tooltool builds of sccache onto various containers and systems. Additionally a new `rust-lang-ci-sccache` bucket is configured to hold all of our ccache goodies. --- Note that this does not currently change Windows [due to previously written up issues](#38119 (comment)). Despite that, however, I was curious to get timings for the builds on Travis to see what ranges we're working with. As a result, this is a WIP PR I'm using to gauge build times and such.
This commit adds support for sccache, a ccache-like compiler which works on MSVC
and stores results into an S3 bucket. This also switches over all Travis and
AppVeyor automation to using sccache to ensure a shared and unified cache over
time which can be shared across builders.
The support for sccache manifests as a new
--enable-sccacheoption whichinstructs us to configure LLVM differently to use a 'sccache' binary instead of
a 'ccache' binary. All docker images for Travis builds are updated to download
Mozilla's tooltool builds of sccache onto various containers and systems.
Additionally a new
rust-lang-ci-sccachebucket is configured to hold all ofour ccache goodies.
Note that this does not currently change Windows due to previously written up issues. Despite that, however, I was curious to get timings for the builds on Travis to see what ranges we're working with. As a result, this is a WIP PR I'm using to gauge build times and such.