Make cargo a workspace #11851

weihanglo · 2023-03-14T16:54:58Z

What does this PR try to resolve?

The first step of making cargo a workspace.

Benefits:

Dogfooding ourselves.
Unblock [BLOCKED] upgrade gix to v0.41 to fix flaky auth tests. #11831: It got stuck because the new version of tempfile using windows-sys but some issues haven't yet be solved in rust-lang/rust.
Make cargo xtask or similar developer workflow possible (e.g., src/doc/build-man.sh is not able to run on platforms missing sh #11717)
Having our own Cargo.lock, so our CI can cover the exact binary going to ship. Also free Cargo from CI breaks due to dependency patch releases.
Probably more? Please add them by yourself.

How should we test and review this PR?

Please review it commit by commit. A companion PR is here rust-lang/rust#109133, and should be reviewed together.

Unresolved issues

To limit the scope of this pull request, the following issues are intentionally left unresolved. They will be addressed right after this pull request gets merged.

Make benches/capture and benches/capture workspace members. (Addressed with 2cf9718)
Make crates/resolver-tests a workspace member. (Addressed with Update proptest #11886)
~~Fix clippy warnings and re-enable clippy check in CI for all workspace members.~~
- Blocked on Add a [lints] table to Cargo.toml rfcs#3389 so we can more easily propagate our clippy settings
Fix rustdoc warnings and re-enable rustdoc check in CI for all workspace members.
Fix linkchecker.sh warnings in CI (Make cargo a workspace #11851 (comment))
Leverage workspace flag --workspace when running cargo build or cargo test, instead of using flag -p.
Leverage glob syntax when probing members in [workspace] in Cargo.toml (i.e., crates/*).

Additional information

This depends on prior works from @Muscraft and @ehuss. Credits to them!

rustbot · 2023-03-14T16:55:04Z

r? @epage

(rustbot has picked a reviewer for you, use r? to override)

epage · 2023-03-14T17:10:13Z

Should benches/capture, benches/capture, crates/resolver-tests be a part of the workspace?

imo everything buildable should be in it

Should we fix clippy warning in this pull request or in a follow-up one? I tend to the latter

I agree that it should be separate. Sometimes, I'll actually do it before by making a commit on top that fixes them but then cherry-pick that into a separate branch. However, I'm not saying we have to do this and we can instead pick it up later

.github/workflows/main.yml

Cargo.toml

This is to help prepare for checking for doc warnings across the entire workspace being created in rust-lang#11851 `Mutation` was made `pub`, along with its fields, but they aren't actually usable with anything, so I went and made it private to match what its documentation references

docs: Address warnings This is to help prepare for checking for doc warnings across the entire workspace being created in #11851 `Mutation` was made `pub`, along with its fields, but they aren't actually usable with anything, so I went and made it private to match what its documentation references

Cargo.lock

Cargo.toml

.github/workflows/main.yml

Cargo.toml

ehuss · 2023-03-24T02:15:07Z

Cargo.toml

+  # For linkchecker (downloaded during CI) and semver-check
+  # TODO: consider move semver-check to crates/ folder


Does the linkcheck.sh script need to be updated to support running from a directory separate from the book?

It definitely could. I just postpone it a bit to make this PR easier to get merged. I'll add this suggestion to TODO as well.

.github/workflows/main.yml

Some dependencies in `resolver-tests` do not have any license information. This prevent it from being a member when integrating in rust-lang/rust. Will figure it out after. Co-authored-by: Scott Schafer <schaferjscott@gmail.com> Co-authored-by: Eric Huss <eric@huss.org>

This is primarily for the release process of rust-lang/rust. Note that in rustc-worksace-hack[1] it enable http2 via libnghttp2, cargo probably needs to enable it to compile in rust-lang/rust. [1]: https://github.com/rust-lang/rust/blob/992d154f3a84cc8abcefcf6e6cf3698e4821b506/src/tools/rustc-workspace-hack/Cargo.toml#L77 Co-authored-by: Scott Schafer <schaferjscott@gmail.com> Co-authored-by: Eric Huss <eric@huss.org>

Co-authored-by: Scott Schafer <schaferjscott@gmail.com> Co-authored-by: Eric Huss <eric@huss.org>

weihanglo · 2023-04-12T13:01:34Z

@ehuss, I guess we get a green light from t-bootstrap (rust-lang/rust#109133 (comment)). Do we want to move forward?

ehuss · 2023-04-12T16:47:26Z

I'm a little concerned about trying to land this just before the beta branch. I suspect this will be a pretty major change, and may have a high risk that it will encounter CI issues or other regressions we'll need to deal with. Would you mind waiting a few days until that happens?

And, BTW, would you mind doing a submodule update todoy or tomorrow to include #11960 so we don't need to backport it after the beta branch? I can do it if needed. And also maybe consider #11878?

weihanglo · 2023-04-12T17:48:08Z

Sure for all. I'll take a look at #11878 today.

ehuss · 2023-04-15T18:05:35Z

@weihanglo It looks like the bump has landed. Whenever you are ready, feel free to r=me this PR and the rust-lang/rust one as well.

weihanglo · 2023-04-15T20:24:02Z

Thanks!

@bors r=ehuss

bors · 2023-04-15T20:24:04Z

📌 Commit aaca5a0 has been approved by ehuss

It is now in the queue for this repository.

bors · 2023-04-15T20:24:18Z

⌛ Testing commit aaca5a0 with merge 39116cc...

bors · 2023-04-15T21:06:29Z

☀️ Test successful - checks-actions
Approved by: ehuss
Pushing 39116cc to master...

…ehuss Make cargo a workspace 8 commits in 7bf43f028ba5eb1f4d70d271c2546c38512c9875..39116ccc9b420a883a98a960f0597f9cf87414b8 2023-04-10 16:01:41 +0000 to 2023-04-15 20:24:15 +0000 - Make cargo a workspace (rust-lang/cargo#11851) - Fix flaky not_found_permutations test. (rust-lang/cargo#11976) - Use restricted Damerau-Levenshtein algorithm (rust-lang/cargo#11963) - Correct the bug report for `cargo clippy --fix` (rust-lang/cargo#11882) - Stabilize `cargo logout` (rust-lang/cargo#11950) - Add more information to HTTP errors to help with debugging. (rust-lang/cargo#11878) - Use registry.default for login/logout (rust-lang/cargo#11949) - Change -C to be unstable (rust-lang/cargo#11960) --- ### What does this PR try to resolve? Making cargo a workspace. Why doing this? * `rustc-workspace-hack` is primarily for sharing dependencies between rls and cargo, as rls previously depends on cargo. After rls retired, it is no longer the case sharing dependencies. * It's q bit painful that cargo needs to deal with some dependency and licensing complexities. For example, rust-lang#108665 failed because of the interaction bewteen `windows-sys` and `raw-dylib`. It currenctly blocks cargo's feature `-Zgitxodie` from moving forward. * See rust-lang/cargo#11851 ### Benchmark result I've done a simple benchmark on both keeping or removing entire `rustc-workspace-hack`. It had no significant difference. Both took ~2m30s to finish `./x.py build -j8 src/tools/cargo src/tools/rls src/tools/clippy src/tools/miri src/tools/rustfmt`. Environment info: ``` host: aarch64-apple-darwin os: Mac OS 13.2.1 [64-bit] ``` A sophisticated benchmark may be needed. ### Additional information This depends on prior works from `@Muscraft` and `@ehuss.` Credits to them!

Since rust-lang/cargo#11851, Cargo became a Cargo workspace of itself. However, since `src/tools/linkchecker` cannot run inside a workspace, Cargo needs a workaround that excludes `src/doc` from workspace member probing. To remove this hack, this PR adds a new optional argument `--path` for `linkchecker.sh`. With this new argument, `linkchecker.sh` can be run from a directory separate from the book. This also benefits other projects using linkchecker, as they can run it under target directory or any other directory, reducing leftover.

linkchecker: running from a directory separate from the book Since rust-lang/cargo#11851, Cargo became a Cargo workspace of itself. However, since `src/tools/linkchecker` cannot run inside a workspace, Cargo needs a workaround that excludes `src/doc` from workspace member probing. To remove this hack, this PR adds a new optional argument `--path` for `linkchecker.sh`. With this new argument, `linkchecker.sh` can be run from a directory separate from the book. This also benefits other projects using linkchecker, as they can run it under target directory or any other directory, reducing leftover.

chore: Use `[workspace.dependencies]` ### What does this PR try to resolve? Keeping shared dependencies in sync. It does this by utilizing workspace inheritance and adding `[workspace.dependencies]`. ### Additional information The ability to use workspace inheritance was added in #11851, as that PR added a workspace for `cargo`. The changes were made by a tool I wrote to assist in finding duplicate dependencies.

rustbot assigned epage Mar 14, 2023

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 14, 2023

epage reviewed Mar 14, 2023

View reviewed changes

weihanglo mentioned this pull request Mar 14, 2023

Make cargo a workspace rust-lang/rust#109133

Merged

weihanglo force-pushed the make-cargo-a-workspace branch 2 times, most recently from db3a838 to 712e7af Compare March 15, 2023 12:07

epage approved these changes Mar 15, 2023

View reviewed changes

epage mentioned this pull request Mar 15, 2023

docs: Address warnings #11856

Merged

weihanglo force-pushed the make-cargo-a-workspace branch from 712e7af to 54810df Compare March 17, 2023 13:07

ehuss reviewed Mar 18, 2023

View reviewed changes

Cargo.lock Outdated Show resolved Hide resolved

Cargo.toml Outdated Show resolved Hide resolved

weihanglo force-pushed the make-cargo-a-workspace branch from 54810df to b36c0b2 Compare March 19, 2023 03:55

weihanglo marked this pull request as ready for review March 19, 2023 03:55

Rustin170506 mentioned this pull request Mar 23, 2023

Make clippy happy #11867

Closed

weihanglo force-pushed the make-cargo-a-workspace branch 2 times, most recently from 68399a1 to f725d3d Compare March 23, 2023 09:48

Muscraft reviewed Mar 23, 2023

View reviewed changes

.github/workflows/main.yml Outdated Show resolved Hide resolved

ehuss reviewed Mar 24, 2023

View reviewed changes

ehuss reviewed Mar 26, 2023

View reviewed changes

.github/workflows/main.yml Outdated Show resolved Hide resolved

weihanglo force-pushed the make-cargo-a-workspace branch from 9eed0c5 to 80d4306 Compare March 26, 2023 23:28

weihanglo and others added 5 commits April 12, 2023 11:30

ci: use -p to specify which package to build an test

36f00f4

Co-authored-by: Scott Schafer <schaferjscott@gmail.com> Co-authored-by: Eric Huss <eric@huss.org>

chore: track Cargo.lock

0289f2b

Co-authored-by: Scott Schafer <schaferjscott@gmail.com> Co-authored-by: Eric Huss <eric@huss.org>

chore: remove dependency rustc-workspace-hack

aaca5a0

Co-authored-by: Scott Schafer <schaferjscott@gmail.com> Co-authored-by: Eric Huss <eric@huss.org>

weihanglo force-pushed the make-cargo-a-workspace branch from 80d4306 to aaca5a0 Compare April 12, 2023 10:33

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 15, 2023

bors merged commit 39116cc into rust-lang:master Apr 15, 2023

weihanglo deleted the make-cargo-a-workspace branch April 15, 2023 21:13

ehuss added this to the 1.71.0 milestone Apr 17, 2023

This was referenced Apr 17, 2023

Leverage the --workspace flag when running cargo command against cargo itself #11987

Closed

Check if a pull request forgot to commit Cargo.lock in CI #11992

Closed

weihanglo mentioned this pull request Apr 19, 2023

linkchecker: running from a directory separate from the book rust-lang/rust#110565

Merged

Muscraft mentioned this pull request Apr 28, 2023

chore: Use [workspace.dependencies] #12057

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make cargo a workspace #11851

Make cargo a workspace #11851

weihanglo commented Mar 14, 2023 •

edited

Loading

rustbot commented Mar 14, 2023

epage commented Mar 14, 2023

ehuss Mar 24, 2023

weihanglo Mar 24, 2023

weihanglo commented Apr 12, 2023

ehuss commented Apr 12, 2023

weihanglo commented Apr 12, 2023

ehuss commented Apr 15, 2023

weihanglo commented Apr 15, 2023

bors commented Apr 15, 2023

bors commented Apr 15, 2023

bors commented Apr 15, 2023

		# For linkchecker (downloaded during CI) and semver-check
		# TODO: consider move semver-check to crates/ folder

Make cargo a workspace #11851

Make cargo a workspace #11851

Conversation

weihanglo commented Mar 14, 2023 • edited Loading

What does this PR try to resolve?

How should we test and review this PR?

Unresolved issues

Additional information

rustbot commented Mar 14, 2023

epage commented Mar 14, 2023

ehuss Mar 24, 2023

Choose a reason for hiding this comment

weihanglo Mar 24, 2023

Choose a reason for hiding this comment

weihanglo commented Apr 12, 2023

ehuss commented Apr 12, 2023

weihanglo commented Apr 12, 2023

ehuss commented Apr 15, 2023

weihanglo commented Apr 15, 2023

bors commented Apr 15, 2023

bors commented Apr 15, 2023

bors commented Apr 15, 2023

weihanglo commented Mar 14, 2023 •

edited

Loading