compiletest: Introduce `// {check,build,run}-pass` pass modes #61778

petrochenkov · 2019-06-12T16:06:00Z

Pass UI tests now have three modes

// check-pass
// build-pass
// run-pass

mirroring equivalent well-known cargo commands.

// check-pass will compile the test skipping codegen (which is expensive and isn't supposed to fail in most cases).
// build-pass will compile and link the test without running it.
// run-pass will compile, link and run the test.
Tests without a "pass" annotation are still considered "fail" tests.

Most UI tests would probably want to switch to check-pass.
Tests validating codegen would probably want to run the generated code as well and use run-pass.
build-pass should probably be rare (linking tests?).

#61755 will provide a way to run the tests with any mode, e.g. bump check-pass tests to run-pass to satisfy especially suspicious people, and be able to make sure that codegen doesn't breaks in some entirely unexpected way.
Tests marked with any mode are expected to pass with any other mode, if that's not the case for some legitimate reason, then the test should be made a "fail" test rather than a "pass" test.
Perhaps some secondary CI can verify this invariant, but that's not super urgent.

// compile-pass still works and is equivalent to build-pass.
Why is // compile-pass bad - 1) it gives an impression that the test is only compiled, but not linked, 2) it doesn't mirror a cargo command.
It can be removed some time in the future in a separate PR.

cc #61712

rust-highfive · 2019-06-12T16:06:12Z

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

petrochenkov · 2019-06-12T16:06:45Z

r? @Mark-Simulacrum @Centril
cc @rust-lang/compiler

Centril · 2019-06-12T16:13:01Z

Tests without a "pass" annotation are still considered "fail" tests.

I would eventually like to have mandatory // compile-fail annotations for easier grepping but that is a larger task.

Mark-Simulacrum · 2019-06-12T16:22:39Z

It feels like in order for #61755 to work well we'd want to actually require that all tests are annotated with all three "modes" (check, build, run), since presumably you could have a test that checks successfully, builds successfully, but should fail when being run. There could be some heuristic that run-pass implies check-pass and build-pass (and build-pass implies check-pass). Otherwise we'd not be able to effectively switch all tests to --stop-at run or whatever flag we choose in #61755, as the test suite would likely fail.

Other than these concerns r=me on the actual implementation though.

petrochenkov · 2019-06-12T16:23:57Z

@Centril

compile-fail annotations for easier grepping

grep -L?

It's nice to have tests without extra noise.
For example, most (non-run-pass) UI tests want #![allow(unused)] to suppress unused warnings irrelevant to the test's point, so #![allow(unused)] is moved into compiletest and enabled by default, so it doesn't have to be added to individual tests.
No annotation for compile-fail seems to be a good default too.

petrochenkov · 2019-06-12T16:28:32Z

@Mark-Simulacrum

you could have a test that checks successfully, builds successfully, but should fail when being run.

Such tests will need to be moved to the run-fail test suit.
(Which could be eventually merged into ui as well.)
That's the general idea - "fails at any stage -> not a pass test".

(Tests failing with --stop-at run will need to be audited though, I'm pretty sure a number of them will fail for silly reasons not related to the test's purpose in any way.)

Mark-Simulacrum · 2019-06-12T16:29:42Z

Ah, I thought we had already merged the run-fail suite in, okay.

src/test/ui/coherence/coherence-projection-ok-orphan.rs

src/test/ui/consts/const-eval/const_transmute.rs

src/tools/compiletest/src/header.rs

Centril · 2019-06-12T16:39:15Z

grep -L?

It's nice to have tests without extra noise.
For example, most UI tests want #![allow(unused)] to suppress unused warnings irrelevant to the test's point, so #![allow(unused)] is moved into compiletest and enabled by default, so it doesn't have to be added to individual tests.
No annotation for compile-fail seems to be a good default too.

I suppose it's a question wrt. noise/signal. I think it's more robust wrt. change to do // compile-fail specifically so that tooling (e.g. wg-grammar stuff) doesn't need to suss out various implicit meanings of various compiletest attributes. It also makes the test suite more easy to understand for beginners.

I think fn main() {} is more annoying personally... :)

Centril · 2019-06-15T18:30:10Z

Would be good to land this so we can unblock --pass check... :)

petrochenkov · 2019-06-23T11:51:32Z

@bors r=Mark-Simulacrum

bors · 2019-06-23T11:51:33Z

📌 Commit 0886bc4 has been approved by Mark-Simulacrum

compiletest: Introduce `// {check,build,run}-pass` pass modes Pass UI tests now have three modes ``` // check-pass // build-pass // run-pass ``` mirroring equivalent well-known `cargo` commands. `// check-pass` will compile the test skipping codegen (which is expensive and isn't supposed to fail in most cases). `// build-pass` will compile and link the test without running it. `// run-pass` will compile, link and run the test. Tests without a "pass" annotation are still considered "fail" tests. Most UI tests would probably want to switch to `check-pass`. Tests validating codegen would probably want to run the generated code as well and use `run-pass`. `build-pass` should probably be rare (linking tests?). rust-lang#61755 will provide a way to run the tests with any mode, e.g. bump `check-pass` tests to `run-pass` to satisfy especially suspicious people, and be able to make sure that codegen doesn't breaks in some entirely unexpected way. Tests marked with any mode are expected to pass with any other mode, if that's not the case for some legitimate reason, then the test should be made a "fail" test rather than a "pass" test. Perhaps some secondary CI can verify this invariant, but that's not super urgent. `// compile-pass` still works and is equivalent to `build-pass`. Why is `// compile-pass` bad - 1) it gives an impression that the test is only compiled, but not linked, 2) it doesn't mirror a cargo command. It can be removed some time in the future in a separate PR. cc rust-lang#61712

@ghost

Rollup of 5 pull requests Successful merges: - #61778 (compiletest: Introduce `// {check,build,run}-pass` pass modes) - #62037 (Speed up tidy) - #62052 (submodules: Update clippy from 5a11ed7 to c5d1ecd) - #62070 (Run rustfmt on some libsyntax files) - #62075 (Remove `ast::Guard`) Failed merges: r? @ghost

bors · 2019-06-23T17:16:29Z

⌛ Testing commit 0886bc4 with merge 2cd5ed4...

compiletest: Introduce `// {check,build,run}-pass` pass modes Pass UI tests now have three modes ``` // check-pass // build-pass // run-pass ``` mirroring equivalent well-known `cargo` commands. `// check-pass` will compile the test skipping codegen (which is expensive and isn't supposed to fail in most cases). `// build-pass` will compile and link the test without running it. `// run-pass` will compile, link and run the test. Tests without a "pass" annotation are still considered "fail" tests. Most UI tests would probably want to switch to `check-pass`. Tests validating codegen would probably want to run the generated code as well and use `run-pass`. `build-pass` should probably be rare (linking tests?). #61755 will provide a way to run the tests with any mode, e.g. bump `check-pass` tests to `run-pass` to satisfy especially suspicious people, and be able to make sure that codegen doesn't breaks in some entirely unexpected way. Tests marked with any mode are expected to pass with any other mode, if that's not the case for some legitimate reason, then the test should be made a "fail" test rather than a "pass" test. Perhaps some secondary CI can verify this invariant, but that's not super urgent. `// compile-pass` still works and is equivalent to `build-pass`. Why is `// compile-pass` bad - 1) it gives an impression that the test is only compiled, but not linked, 2) it doesn't mirror a cargo command. It can be removed some time in the future in a separate PR. cc #61712

bors · 2019-06-23T20:04:57Z

☀️ Test successful - checks-travis, status-appveyor
Approved by: Mark-Simulacrum
Pushing 2cd5ed4 to master...

@petrochenkov

…petrochenkov Add `--pass $mode` to compiletest through `./x.py` Adds a flag `--pass $mode` to compiletest, which is exposed through `./x.py`. When `--pass $mode` is passed, `{check,build,compile,run}-pass` tests will be forced to run under the given `$mode` unless the directive `// ignore-pass` exists in the test file. The modes are explained in rust-lang#61778: - `check` has the same effect as `cargo check` - `build` or `compile` have the same effect as `cargo build` - `run` has the same effect as `cargo run` On my machine, `./x.py -i test src/test/run-pass --stage 1 --pass check` takes 38 seconds whereas it takes 2 min 7 seconds without `--pass check`. cc rust-lang#61712 r? @petrochenkov

@petrochenkov

…petrochenkov Add `--pass $mode` to compiletest through `./x.py` Adds a flag `--pass $mode` to compiletest, which is exposed through `./x.py`. When `--pass $mode` is passed, `{check,build,compile,run}-pass` tests will be forced to run under the given `$mode` unless the directive `// ignore-pass` exists in the test file. The modes are explained in rust-lang#61778: - `check` has the same effect as `cargo check` - `build` or `compile` have the same effect as `cargo build` - `run` has the same effect as `cargo run` On my machine, `./x.py -i test src/test/run-pass --stage 1 --pass check` takes 38 seconds whereas it takes 2 min 7 seconds without `--pass check`. cc rust-lang#61712 r? @petrochenkov

@petrochenkov

…petrochenkov Add `--pass $mode` to compiletest through `./x.py` Adds a flag `--pass $mode` to compiletest, which is exposed through `./x.py`. When `--pass $mode` is passed, `{check,build,compile,run}-pass` tests will be forced to run under the given `$mode` unless the directive `// ignore-pass` exists in the test file. The modes are explained in rust-lang#61778: - `check` has the same effect as `cargo check` - `build` or `compile` have the same effect as `cargo build` - `run` has the same effect as `cargo run` On my machine, `./x.py -i test src/test/run-pass --stage 1 --pass check` takes 38 seconds whereas it takes 2 min 7 seconds without `--pass check`. cc rust-lang#61712 r? @petrochenkov

@petrochenkov

…petrochenkov Add `--pass $mode` to compiletest through `./x.py` Adds a flag `--pass $mode` to compiletest, which is exposed through `./x.py`. When `--pass $mode` is passed, `{check,build,compile,run}-pass` tests will be forced to run under the given `$mode` unless the directive `// ignore-pass` exists in the test file. The modes are explained in rust-lang#61778: - `check` has the same effect as `cargo check` - `build` or `compile` have the same effect as `cargo build` - `run` has the same effect as `cargo run` On my machine, `./x.py -i test src/test/run-pass --stage 1 --pass check` takes 38 seconds whereas it takes 2 min 7 seconds without `--pass check`. cc rust-lang#61712 r? @petrochenkov

RalfJung · 2019-06-29T09:30:57Z

// compile-pass still works and is equivalent to build-pass.
Why is // compile-pass bad - 1) it gives an impression that the test is only compiled, but not linked, 2) it doesn't mirror a cargo command.
It can be removed some time in the future in a separate PR.

Is there an issue tracking deprecation/removal of compile-pass?

Centril · 2019-06-29T09:33:01Z

Is there an issue tracking deprecation/removal of compile-pass?

Not to my knowledge but it should be a rather trivial find/replace with // compile-pass to // build-pass.

cramertj · 2019-07-01T16:51:10Z

@Centril I'd figure most examples would want to be // compile-pass -> // check-pass, not // build-pass.

Centril · 2019-07-01T16:53:28Z

@cramertj Yes, but it's a risky move to blanket replace // compile-pass with // check-pass since that does something different and may reduce test coverage.

cramertj · 2019-07-01T16:55:54Z

yeah, just pointing out that we might not want to blanket search-and-replace to // build-pass, since we'd no longer have a way to tell which things were intentionally build-pass or check-pass vs. things that just hadn't been moved over yet. We could solve that by adding a note of some kind in the output, e.g. // build-pass (unreviewed, possibly check-pass)

Centril · 2019-07-01T16:58:32Z

We could solve that by adding a note of some kind in the output, e.g. // build-pass (unreviewed, possibly check-pass)

@cramertj This seems fine; want to file a C-cleanup issue?

cramertj · 2019-07-01T18:33:37Z

Opened #62277.

rust-highfive assigned nikomatsakis Jun 12, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 12, 2019

rust-highfive assigned Mark-Simulacrum and unassigned nikomatsakis Jun 12, 2019

petrochenkov mentioned this pull request Jun 12, 2019

Add --pass $mode to compiletest through ./x.py #61755

Merged

Centril reviewed Jun 12, 2019

View reviewed changes

src/test/ui/coherence/coherence-projection-ok-orphan.rs Show resolved Hide resolved

src/test/ui/consts/const-eval/const_transmute.rs Show resolved Hide resolved

src/tools/compiletest/src/header.rs Outdated Show resolved Hide resolved

petrochenkov added 4 commits June 16, 2019 12:23

compiletest: Introduce // {check,build,run}-pass pass modes

6203f68

compiletest: Validate pass modes harder

8e8fba1

compiletest: Remove skip-codegen

932ea64

compiletest: Move pass mode update into a separate function

0886bc4

petrochenkov force-pushed the pass branch from 3ed9b8a to 0886bc4 Compare June 16, 2019 09:45

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 23, 2019

Centril mentioned this pull request Jun 23, 2019

Rollup of 5 pull requests #62080

Closed

bors added the merged-by-bors This PR was explicitly merged by bors. label Jun 23, 2019

bors merged commit 0886bc4 into rust-lang:master Jun 23, 2019

bors mentioned this pull request Jun 23, 2019

Normalize use of backticks compiler messages #61901

Closed

petrochenkov mentioned this pull request Jun 29, 2019

More systematic compile/run markup for "pass" UI tests #61712

Closed

cramertj mentioned this pull request Jul 1, 2019

Move compile-pass tests to check-pass or build-pass #62277

Open

petrochenkov deleted the pass branch July 1, 2019 22:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

compiletest: Introduce `// {check,build,run}-pass` pass modes #61778

compiletest: Introduce `// {check,build,run}-pass` pass modes #61778

petrochenkov commented Jun 12, 2019 •

edited

Loading

rust-highfive commented Jun 12, 2019

petrochenkov commented Jun 12, 2019 •

edited

Loading

Centril commented Jun 12, 2019

Mark-Simulacrum commented Jun 12, 2019

petrochenkov commented Jun 12, 2019 •

edited

Loading

petrochenkov commented Jun 12, 2019 •

edited

Loading

Mark-Simulacrum commented Jun 12, 2019

Centril commented Jun 12, 2019

Centril commented Jun 15, 2019

petrochenkov commented Jun 23, 2019

bors commented Jun 23, 2019

bors commented Jun 23, 2019

bors commented Jun 23, 2019

RalfJung commented Jun 29, 2019

Centril commented Jun 29, 2019

cramertj commented Jul 1, 2019

Centril commented Jul 1, 2019

cramertj commented Jul 1, 2019

Centril commented Jul 1, 2019

cramertj commented Jul 1, 2019

compiletest: Introduce // {check,build,run}-pass pass modes #61778

compiletest: Introduce // {check,build,run}-pass pass modes #61778

Conversation

petrochenkov commented Jun 12, 2019 • edited Loading

rust-highfive commented Jun 12, 2019

petrochenkov commented Jun 12, 2019 • edited Loading

Centril commented Jun 12, 2019

Mark-Simulacrum commented Jun 12, 2019

petrochenkov commented Jun 12, 2019 • edited Loading

petrochenkov commented Jun 12, 2019 • edited Loading

Mark-Simulacrum commented Jun 12, 2019

Centril commented Jun 12, 2019

Centril commented Jun 15, 2019

petrochenkov commented Jun 23, 2019

bors commented Jun 23, 2019

bors commented Jun 23, 2019

bors commented Jun 23, 2019

RalfJung commented Jun 29, 2019

Centril commented Jun 29, 2019

cramertj commented Jul 1, 2019

Centril commented Jul 1, 2019

cramertj commented Jul 1, 2019

Centril commented Jul 1, 2019

cramertj commented Jul 1, 2019

compiletest: Introduce `// {check,build,run}-pass` pass modes #61778

compiletest: Introduce `// {check,build,run}-pass` pass modes #61778

petrochenkov commented Jun 12, 2019 •

edited

Loading

petrochenkov commented Jun 12, 2019 •

edited

Loading

petrochenkov commented Jun 12, 2019 •

edited

Loading

petrochenkov commented Jun 12, 2019 •

edited

Loading