build dist for x86_64-unknown-illumos

[jclulow]

This change creates a new Docker image, "dist-x86_64-illumos", and sets
things up to build the full set of "dist" packages for illumos hosts, so
that illumos users can use "rustup" to install packages. It also
adjusts the manifest builder to expect complete toolchains for this
platform.

[rust-highfive]

r? @pietroalbini

(rust_highfive has picked a reviewer for you, use r? to override)

[jclulow]

Note that I am not super familiar with the CI/infra stuff at play here, and have mostly just tried to make everything look as much like it does for FreeBSD as possible, on the basis that it is a similarly cross-compiled target. If there is anybody in the infra team to whom I should speak about this as part of getting review, please let me know! Thanks.

I have tested the Docker CI image a bunch of times locally, and @pfmooney and I are using illumos toolchain bits built this way as we work on platform support in various ecosystem crates. Things have been going well thus far!

[jclulow]

A further note: this makes use of an archive of core OS artefacts from an illumos system, which is described in some more detail over at: https://github.com/illumos/sysroot

[rust-highfive]

The job mingw-check of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

2020-04-18T06:56:45.4367914Z ========================== Starting Command Output ===========================
2020-04-18T06:56:45.4372176Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/39c18008-e123-4505-b4a0-f32cdf4783a5.sh
2020-04-18T06:56:45.4372608Z 
2020-04-18T06:56:45.4379315Z ##[section]Finishing: Disable git automatic line ending conversion
2020-04-18T06:56:45.4410142Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/71272/merge to s
2020-04-18T06:56:45.4415299Z Task         : Get sources
2020-04-18T06:56:45.4415821Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-04-18T06:56:45.4416311Z Version      : 1.0.0
2020-04-18T06:56:45.4416652Z Author       : Microsoft
---
2020-04-18T06:56:46.4479943Z ##[command]git remote add origin https://github.com/rust-lang/rust
2020-04-18T06:56:46.4484154Z ##[command]git config gc.auto 0
2020-04-18T06:56:46.4486932Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2020-04-18T06:56:46.4489483Z ##[command]git config --get-all http.proxy
2020-04-18T06:56:46.4494805Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/71272/merge:refs/remotes/pull/71272/merge
---
2020-04-18T06:59:23.0984905Z  ---> 78ad2f4d4aca
2020-04-18T06:59:23.0985122Z Step 6/7 : ENV RUN_CHECK_WITH_PARALLEL_QUERIES 1
2020-04-18T06:59:23.0985470Z  ---> Using cache
2020-04-18T06:59:23.0985791Z  ---> 4d2dc61c4d00
2020-04-18T06:59:23.0987018Z Step 7/7 : ENV SCRIPT python3 ../x.py test src/tools/expand-yaml-anchors &&            python3 ../x.py check --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu &&            python3 ../x.py build --stage 0 src/tools/build-manifest &&            python3 ../x.py test --stage 0 src/tools/compiletest &&            python3 ../x.py test src/tools/tidy &&            /scripts/validate-toolstate.sh
2020-04-18T06:59:23.0988214Z  ---> 776b6266a8b7
2020-04-18T06:59:23.0988404Z Successfully built 776b6266a8b7
2020-04-18T06:59:23.1033727Z Successfully tagged rust-ci:latest
2020-04-18T06:59:23.1327714Z Built container sha256:776b6266a8b7d63e2d3c2b5a784dbf521184a904fb10bf818c6b5c7e1ab74d4a
2020-04-18T06:59:23.1327714Z Built container sha256:776b6266a8b7d63e2d3c2b5a784dbf521184a904fb10bf818c6b5c7e1ab74d4a
2020-04-18T06:59:23.1342067Z Looks like docker image is the same as before, not uploading
2020-04-18T06:59:23.8278081Z [CI_JOB_NAME=mingw-check]
2020-04-18T06:59:23.8471540Z [CI_JOB_NAME=mingw-check]
2020-04-18T06:59:23.8510171Z == clock drift check ==
2020-04-18T06:59:23.8519390Z   local time: Sat Apr 18 06:59:23 UTC 2020
2020-04-18T06:59:24.0299077Z   network time: Sat, 18 Apr 2020 06:59:24 GMT
2020-04-18T06:59:24.0326120Z Starting sccache server...
2020-04-18T06:59:24.1339072Z configure: processing command line
2020-04-18T06:59:24.1340436Z configure: 
2020-04-18T06:59:24.1341354Z configure: rust.parallel-compiler := True
---
2020-04-18T07:02:36.5725732Z     Checking rustc_feature v0.0.0 (/checkout/src/librustc_feature)
2020-04-18T07:02:36.6041449Z     Checking fmt_macros v0.0.0 (/checkout/src/libfmt_macros)
2020-04-18T07:02:36.7624008Z     Checking rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
2020-04-18T07:02:36.9094498Z     Checking rustc_hir v0.0.0 (/checkout/src/librustc_hir)
2020-04-18T07:02:37.3463785Z     Checking rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
2020-04-18T07:02:39.3017365Z     Checking rustc_attr v0.0.0 (/checkout/src/librustc_attr)
2020-04-18T07:02:39.6833707Z     Checking rustc_parse v0.0.0 (/checkout/src/librustc_parse)
2020-04-18T07:02:41.3752156Z     Checking rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
2020-04-18T07:02:41.7372924Z     Checking rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
2020-04-18T07:04:10.6104043Z configure: rust.debug-assertions := True
2020-04-18T07:04:10.6104415Z configure: build.cargo-native-static := True
2020-04-18T07:04:10.6104639Z configure: rust.channel         := nightly
2020-04-18T07:04:10.6104853Z configure: build.submodules     := False
2020-04-18T07:04:10.6105305Z configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
2020-04-18T07:04:10.6105704Z configure: writing `config.toml` in current directory
2020-04-18T07:04:10.6105893Z configure: 
2020-04-18T07:04:10.6106195Z configure: run `python /checkout/x.py --help`
2020-04-18T07:04:10.6106360Z configure: 
---
2020-04-18T07:05:34.0608077Z Hugepagesize:       2048 kB
2020-04-18T07:05:34.0608239Z DirectMap4k:      131008 kB
2020-04-18T07:05:34.0608413Z DirectMap2M:     4063232 kB
2020-04-18T07:05:34.0608574Z DirectMap1G:     5242880 kB
2020-04-18T07:05:34.0650519Z + python3 ../x.py test src/tools/expand-yaml-anchors
2020-04-18T07:05:35.2103354Z Ensuring the YAML anchors in the GitHub Actions config were expanded
2020-04-18T07:05:35.2103354Z Ensuring the YAML anchors in the GitHub Actions config were expanded
2020-04-18T07:05:35.2139472Z Building stage0 tool expand-yaml-anchors (x86_64-unknown-linux-gnu)
2020-04-18T07:05:35.4127403Z    Compiling unicode-xid v0.2.0
2020-04-18T07:05:35.5302873Z    Compiling syn v1.0.11
2020-04-18T07:05:36.2752363Z    Compiling linked-hash-map v0.5.2
2020-04-18T07:05:36.2971884Z    Compiling lazy_static v1.4.0
2020-04-18T07:05:36.2971884Z    Compiling lazy_static v1.4.0
2020-04-18T07:05:36.4739564Z    Compiling yaml-rust v0.4.3
2020-04-18T07:05:40.0220548Z    Compiling quote v1.0.2
2020-04-18T07:05:51.8362975Z    Compiling thiserror-impl v1.0.5
2020-04-18T07:05:55.6781724Z    Compiling thiserror v1.0.5
2020-04-18T07:05:55.7352002Z    Compiling yaml-merge-keys v0.4.0
2020-04-18T07:05:56.7021699Z    Compiling expand-yaml-anchors v0.1.0 (/checkout/src/tools/expand-yaml-anchors)
2020-04-18T07:05:58.0755267Z     Finished release [optimized] target(s) in 22.86s
2020-04-18T07:05:58.0900984Z error: .github/workflows/ci.yml is not up to date
2020-04-18T07:05:58.0902486Z caused by: src/ci/github-actions/ci.yml and .github/workflows/ci.yml are different
2020-04-18T07:05:58.0903061Z 
2020-04-18T07:05:58.0903061Z 
2020-04-18T07:05:58.0904161Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/expand-yaml-anchors" "check" "/checkout"
2020-04-18T07:05:58.0905222Z 
2020-04-18T07:05:58.0905386Z 
2020-04-18T07:05:58.0905386Z 
2020-04-18T07:05:58.0911568Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/expand-yaml-anchors
2020-04-18T07:05:58.1010381Z == clock drift check ==
2020-04-18T07:05:58.1022959Z   local time: Sat Apr 18 07:05:58 UTC 2020
2020-04-18T07:05:58.1022959Z   local time: Sat Apr 18 07:05:58 UTC 2020
2020-04-18T07:05:58.4081758Z   network time: Sat, 18 Apr 2020 07:05:58 GMT
2020-04-18T07:06:03.9067103Z 
2020-04-18T07:06:03.9067103Z 
2020-04-18T07:06:03.9129276Z ##[error]Bash exited with code '1'.
2020-04-18T07:06:03.9143794Z ##[section]Finishing: Run build
2020-04-18T07:06:03.9189428Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/71272/merge to s
2020-04-18T07:06:03.9193699Z Task         : Get sources
2020-04-18T07:06:03.9193988Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-04-18T07:06:03.9194252Z Version      : 1.0.0
2020-04-18T07:06:03.9194454Z Author       : Microsoft
2020-04-18T07:06:03.9194454Z Author       : Microsoft
2020-04-18T07:06:03.9194752Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
2020-04-18T07:06:03.9195088Z ==============================================================================
2020-04-18T07:06:04.2190874Z Cleaning any cached credential from repository: rust-lang/rust (GitHub)
2020-04-18T07:06:04.2235832Z ##[section]Finishing: Checkout rust-lang/rust@refs/pull/71272/merge to s
2020-04-18T07:06:04.2319020Z Cleaning up task key
2020-04-18T07:06:04.2320158Z Start cleaning up orphan processes.
2020-04-18T07:06:04.2473834Z Terminate orphan process: pid (4448) (python)
2020-04-18T07:06:04.2745514Z ##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

[pietroalbini]

Thanks for the PR! Unfortunately we don't have the CI capacity for an extra builder right now: could you add this target to the dist-various-2 builder?

[jclulow]

Hi, @pietroalbini! As requested, I have moved the build into the existing dist-various-2 builder. I have confirmed that I am able to use the new image locally on my workstation to build a full toolchain for illumos, and the existing standard library archive that is built for Solaris on x86 and SPARC. Please let me know if there is anything else I should look at! Thanks.

[pietroalbini]

Pushed a temporary commit to test the builder in a try build.

@bors try

[bors]

⌛ Trying commit d3c0912c8090af3a2bbf4dd1ab12a8aba4c0aadd with merge c18e4bc243f34aa6702a147a3d961974ed2434c8...

[pietroalbini]

Why was some Solaris code changed as part of this PR?

Also, cc @rust-lang/compiler, this PR practically adds x86_64-unknown-illumos as a Tier 2 target (building a full host compiler). Do you have some requirements on your end for that?

[pfmooney]

Why was some Solaris code changed as part of this PR?

This is to disambiguate illumos and Solaris. Both require a Solaris-y toolchain, but given the divergence in the platforms (see also: #71145 in the effort to split them), they require separate compilers using their respective sysroots.

Also, cc @rust-lang/compiler, this PR practically adds x86_64-unknown-illumos as a Tier 2 target (building a full host compiler). Do you have some requirements on your end for that?

We're interested in making rustup a reality on this platform.

[bors]

💥 Test timed out

[jclulow]

Looking into the failure at https://github.com/rust-lang-ci/rust/runs/605427771 it seems like the --enable-lld flag to Rust's configure may be enabled in error for this image. This is the first time we're actually shipping a toolchain out of dist-various-2, rather than just cross-built library artefacts. Looking at, say, dist-x86_64-freebsd, where a cross built toolchain is delivered, that flag is not present. Indeed, the only other images which include it are dist-x86_64-musl and x86_64-gnu-debug which are probably not (perhaps merely not completely or hygienically) cross built.

I'll investigate what's happening in some more detail, but I think the right thing to do is drop the flag here, because I don't think we need lld and it seems like the cross build for lld is not quite right anyhow.

[nikomatsakis]

Something we've been discussing a lot is that, when we add a new target, we ought to have a list of folks that we can ping for advice, or people that we can request to review PRs that are proposing changes specific to that tier. Another common thing is to have someone who can reproduce and help to diagnose problems specific to that target.

I also think it would make sense to file a Major Change Proposal -- that process is still "underway" but it mostly just means creating an issue with this template.

@jclulow, would you be willing to file a MCP as described above, and include in there the names of any folks who would be willing to help review and maintain the target?

[jclulow]

@jclulow, would you be willing to file a MCP as described above, and include in there the names of any folks who would be willing to help review and maintain the target?

Definitely! @pfmooney and myself in particular are keen to get this across the line in a sustainable way for illumos, and to stick around to assist with maintenance in the future. I'm still working on technical issues with this PR (e.g., #71457), but in the meantime I will write up an MCP! Thanks for pointing us in the right direction.

[bors]

☔ The latest upstream changes (presumably #71434) made this pull request unmergeable. Please resolve the merge conflicts.

[Dylan-DPC-zz]

@jclulow can you rebase? thanks

[jclulow]

@jclulow can you rebase? thanks

Definitely! What should I do with the extra commit that was added on top that seems to make it possible to try this out? Just leave it in place?

[Manishearth]

@bors retry yield

[bors]

⌛ Testing commit 8368a35 with merge 1a3f4791a589d554233d4baf7d94226252d4d15e...

[Manishearth]

@bors retry yield

[Dylan-DPC-zz]

@bors treeclosed-

(will close the tree again if issue persists)

[bors]

⌛ Testing commit 8368a35 with merge 6cc06ce7c6e05afdeb1d3cd3d531547d714adc27...

[Dylan-DPC-zz]

@bors retry
(oops)

[bors]

⌛ Testing commit 8368a35 with merge 89111c88a3537c25d577fa304fbebbc23237fd1e...

[bors]

⌛ Testing commit 8368a35 with merge 66fb914346eb33a90b2945d2dbfbbed978b6c4b3...

[bors]

💥 Test timed out

[Manishearth]

@bors retry

[bors]

⌛ Testing commit 8368a35 with merge c714eae...

[bors]

☀️ Test successful - checks-actions, checks-azure
Approved by: pietroalbini
Pushing c714eae to master...

[liulitchi]

Thank you all, I've just installed Rust on my OpenIndiana OS, and it runs pretty well.
This command should be very helpful for other illumos users:
curl --proto '=https' --tlsv1.2 -sSf https://illumos.org/downloads/rustup/init.sh | bash

[jclulow]

@liulitchi Note that as of rustup 1.23.1 (3df2264a9 2020-11-30) (see also: rust-lang/rustup#2432 and rust-lang/rustup#2580), you should be able to use the stock rustup instead of the preliminary illumos-specific build. The instructions up on https://rustup.rs/ should be sufficient -- if not, open an issue on the rustup repository and tag me in it! Thanks.