Create self-contained directory and move there some of external binaries/libs

[mati865]

One of the steps to reach design described in #68887 (comment)
This PR moves things around and allows link code to handle the new directory structure.

[rust-highfive]

r? @Mark-Simulacrum

(rust_highfive has picked a reviewer for you, use r? to override)

[mati865]

cc @petrochenkov

[Mark-Simulacrum]

I know essentially nothing about these so happy to cede review to @petrochenkov (or someone else). The bootstrap code looks reasonable, though only from a "doesn't do anything unexpected" rather than "does the right thing" perspective.

[petrochenkov]

LGTM.
I can't comment on the cross-compilation case because I'm not too familiar with bootstrap.

[petrochenkov]

I'd like to interject for a moment to paint the bike shed.
I was thinking about choosing a name for the "fallback" / "self-contained" mode and using it consistently.

On one hand this is a "fallback" mode because it's not desirable (also "fallback" is a shorter word).

On the other hand, we can't really fallback from the libraries and objects provided by native toolchain to libraries and objects provided by us, in a sense that "<sysroot>/fallback is always in search paths, but "shadowed" by the native toolchain it it is present".
If gcc is used as a linker, it will hardcode CRT objects without using any search paths, and for libraries there's no way to add our search directory to the end of the list implicitly supplied by gcc.
So we'll have to explicitly use or not use the <sysroot>/fallback directory depending on the mode and if used it will take priority over the native toolchain if it happens to exist after all.

So, I guess, "self-contained" it is.

[mati865]

I don't mind either name.
Once you are happy with the changes and @bors try proves nothing has broke I'll squash the commits.

[mati865]

Included commit moving bundled GCC and Binutils to self-contained dir since it was smaller than I expected. I'll retitle the PR if you are fine with that.

[petrochenkov]

@bors try

[bors]

⌛ Trying commit da03b26cfe80fc12c10d9ed41dd8afe4bdb3484a with merge 8391cc93e571cc4a32d6b0a9958e88a154268c7a...

[bors]

☀️ Try build successful - checks-azure
Build commit: 8391cc93e571cc4a32d6b0a9958e88a154268c7a (8391cc93e571cc4a32d6b0a9958e88a154268c7a)

[petrochenkov]

This distribution layout change may require tweaking manifests used by rustup, for the rust-mingw component in particular.
I'm not sure how to do it and whether it's actually necessary, Mark-Simulacrum may know more, but it all should be in this repository.

[mati865]

This distribution layout change may require tweaking manifests used by rustup, for the rust-mingw component in particular.

I'm not sure what you mean. This works fine when installed with rustup-toolchain-install-master 8391cc93e571cc4a32d6b0a9958e88a154268c7a -c rust-mingw.

Contents of the manifest:

file:lib/rustlib/x86_64-pc-windows-gnu/bin/self-contained/dlltool.exe
file:lib/rustlib/x86_64-pc-windows-gnu/bin/self-contained/GCC-WARNING.txt
file:lib/rustlib/x86_64-pc-windows-gnu/bin/self-contained/ld.exe
file:lib/rustlib/x86_64-pc-windows-gnu/bin/self-contained/libwinpthread-1.dll
file:lib/rustlib/x86_64-pc-windows-gnu/bin/self-contained/x86_64-w64-mingw32-gcc.exe
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libadvapi32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libbcrypt.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libcomctl32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libcomdlg32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libcredui.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libcrypt32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libdbghelp.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libgcc.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libgcc_eh.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libgcc_s.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libgdi32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libiconv.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libimagehlp.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libiphlpapi.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libkernel32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libm.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libmingw32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libmingwex.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libmoldname.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libmsimg32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libmsvcrt.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libodbc32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libole32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/liboleaut32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libopengl32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libpsapi.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libpthread.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/librpcrt4.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libsecur32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libsetupapi.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libshell32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libstdc++.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libsynchronization.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libuser32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libuserenv.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libuuid.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libwinhttp.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libwinmm.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libwinspool.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libws2_32.a
file:lib/rustlib/x86_64-pc-windows-gnu/lib/self-contained/libwsock32.a

[petrochenkov]

It's great if it works without changes, I just mentioned it as something to check and keep in mind.

[mati865]

Okay, this should be ready. Should I squash Address the review commits into previous commits?

[Mark-Simulacrum]

Please do squash the review-addressing commits.

[mati865]

Squashed, rebased and testuite still passes.

[Mark-Simulacrum]

IMO, this is good to r+, but I want to stop before we move ahead and ask -- it feels like this is a pretty major change, and should go through at least a MCP (and maybe even compiler team FCP, given impacts to stable users). But, I don't know much about this area, so I'm going to cede r? @petrochenkov for final approval of the PR in a more conceptual sense (r=me on the literal code changes, though).

[petrochenkov]

Sigh.

@mati865, could you fill the MCP form (https://forge.rust-lang.org/compiler/mcp.html#major-change-proposals-1) and point it to #68887 (comment)?

Then I'll r+ this once someone else sanity checks the idea.

[Mark-Simulacrum]

To be clear if this isn't changing anything all that major, just moving some files around and in a way that shouldn't matter to anyone, then I'm fine moving ahead without that -- I don't really understand our linkage story wrt to self-contained and mingw etc well enough that I can comment well there.

[mati865]

@mati865, could you fill the MCP form (forge.rust-lang.org/compiler/mcp.html#major-change-proposals-1) and point it to #68887 (comment)?

I'll try to prepare it tomorrow.

To be clear if this isn't changing anything all that major, just moving some files around and in a way that shouldn't matter to anyone

The intention is to make no visible changes to the user yet. Few libraries are moved to a subdirectory but changes to src/librustc_codegen_ssa/back/link.rs make it always searched by the linker.
Is the internal structure of rustlib/ considered stable?

[Mark-Simulacrum]

Aha okay, then I'm fine with just @bors r+ this PR, and we can MCP in parallel, before we land user visible changes.

Would be good to get some of the broader strategy written up for rustc dev guide too.

[bors]

📌 Commit 43905cd has been approved by Mark-Simulacrum