-
Notifications
You must be signed in to change notification settings - Fork 12.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Auto merge of #33588 - nikomatsakis:compiletest-ui, r=acrichto
add UI testing framework This adds a framework for capturing and tracking the precise output of rustc, which allows us to check all manner of minor details with the output. It's pretty strict right now -- the output must match almost exactly -- and hence maybe a bit too strict. But I figure we can add wildcards or whatever later. There is also a script intended to make updating the references easy, though the script could make things a *bit* easier (in particular, it'd be nice if it would find the build directory for you automatically). One thing I was wondering about is the best way to test colors. Since windows doesn't embed those in the output stream, this test framework can't test colors on windows -- so I figure we can just write tests that are ignored on windows and which pass `--color=always` or whatever to rustc. cc @jonathandturner r? @alexcrichton
- Loading branch information
Showing
13 changed files
with
354 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
# Guide to the UI Tests | ||
|
||
The UI tests are intended to capture the compiler's complete output, | ||
so that we can test all aspects of the presentation. They work by | ||
compiling a file (e.g., `hello_world/main.rs`), capturing the output, | ||
and then applying some normalization (see below). This normalized | ||
result is then compared against reference files named | ||
`hello_world/main.stderr` and `hello_world/main.stdout`. If either of | ||
those files doesn't exist, the output must be empty. If the test run | ||
fails, we will print out the current output, but it is also saved in | ||
`build/<target-triple>/test/ui/hello_world/main.stdout` (this path is | ||
printed as part of the test failure mesage), so you can run `diff` and | ||
so forth. | ||
|
||
# Editing and updating the reference files | ||
|
||
If you have changed the compiler's output intentionally, or you are | ||
making a new test, you can use the script `update-references.sh` to | ||
update the references. When you run the test framework, it will report | ||
various errors: in those errors is a command you can use to run the | ||
`update-references.sh` script, which will then copy over the files | ||
from the build directory and use them as the new reference. You can | ||
also just run `update-all-references.sh`. In both cases, you can run | ||
the script with `--help` to get a help message. | ||
|
||
# Normalization | ||
|
||
The normalization applied is aimed at filenames: | ||
|
||
- the test directory is replaced with `$DIR` | ||
- all backslashes (\) are converted to forward slashes (/) (for windows) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
// Copyright 2013-2014 The Rust Project Developers. See the COPYRIGHT | ||
// file at the top-level directory of this distribution and at | ||
// http://rust-lang.org/COPYRIGHT. | ||
// | ||
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or | ||
// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license | ||
// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your | ||
// option. This file may not be copied, modified, or distributed | ||
// except according to those terms. | ||
|
||
// Test that compiling hello world succeeds with no output of any kind. | ||
|
||
fn main() { | ||
println!("Hello, world!"); | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
// Copyright 2013-2014 The Rust Project Developers. See the COPYRIGHT | ||
// file at the top-level directory of this distribution and at | ||
// http://rust-lang.org/COPYRIGHT. | ||
// | ||
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or | ||
// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license | ||
// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your | ||
// option. This file may not be copied, modified, or distributed | ||
// except according to those terms. | ||
|
||
// rustc-env:RUST_NEW_ERROR_FORMAT | ||
|
||
fn main() { | ||
let x: u32 = ( | ||
); | ||
} | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
error: mismatched types [--explain E0308] | ||
--> $DIR/main.rs:14:18 | ||
14 |> let x: u32 = ( | ||
|> ^ expected u32, found () | ||
note: expected type `u32` | ||
note: found type `()` | ||
|
||
error: aborting due to previous error |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
#!/bin/bash | ||
# | ||
# Copyright 2015 The Rust Project Developers. See the COPYRIGHT | ||
# file at the top-level directory of this distribution and at | ||
# http://rust-lang.org/COPYRIGHT. | ||
# | ||
# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or | ||
# http://www.apache.org/licenses/LICENSE-2.0> or the MIT license | ||
# <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your | ||
# option. This file may not be copied, modified, or distributed | ||
# except according to those terms. | ||
|
||
# A script to update the references for all tests. The idea is that | ||
# you do a run, which will generate files in the build directory | ||
# containing the (normalized) actual output of the compiler. You then | ||
# run this script, which will copy those files over. If you find | ||
# yourself manually editing a foo.stderr file, you're doing it wrong. | ||
# | ||
# See all `update-references.sh`, if you just want to update a single test. | ||
|
||
if [[ "$1" == "--help" || "$1" == "-h" || "$1" == "" || "$2" != "" ]]; then | ||
echo "usage: $0 <build-directory>" | ||
echo "" | ||
echo "For example:" | ||
echo " $0 ../../../build/x86_64-apple-darwin/test/ui" | ||
fi | ||
|
||
BUILD_DIR=$PWD/$1 | ||
MY_DIR=$(dirname $0) | ||
cd $MY_DIR | ||
find . -name '*.rs' | xargs ./update-references.sh $BUILD_DIR |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,50 @@ | ||
#!/bin/bash | ||
# | ||
# Copyright 2015 The Rust Project Developers. See the COPYRIGHT | ||
# file at the top-level directory of this distribution and at | ||
# http://rust-lang.org/COPYRIGHT. | ||
# | ||
# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or | ||
# http://www.apache.org/licenses/LICENSE-2.0> or the MIT license | ||
# <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your | ||
# option. This file may not be copied, modified, or distributed | ||
# except according to those terms. | ||
|
||
# A script to update the references for particular tests. The idea is | ||
# that you do a run, which will generate files in the build directory | ||
# containing the (normalized) actual output of the compiler. This | ||
# script will then copy that output and replace the "expected output" | ||
# files. You can then commit the changes. | ||
# | ||
# If you find yourself manually editing a foo.stderr file, you're | ||
# doing it wrong. | ||
|
||
if [[ "$1" == "--help" || "$1" == "-h" || "$1" == "" || "$2" == "" ]]; then | ||
echo "usage: $0 <build-directory> <relative-path-to-rs-files>" | ||
echo "" | ||
echo "For example:" | ||
echo " $0 ../../../build/x86_64-apple-darwin/test/ui *.rs */*.rs" | ||
fi | ||
|
||
MYDIR=$(dirname $0) | ||
|
||
BUILD_DIR="$1" | ||
shift | ||
|
||
while [[ "$1" != "" ]]; do | ||
STDERR_NAME="${1/%.rs/.stderr}" | ||
STDOUT_NAME="${1/%.rs/.stdout}" | ||
shift | ||
if [ -f $BUILD_DIR/$STDOUT_NAME ] && \ | ||
! (diff $BUILD_DIR/$STDOUT_NAME $MYDIR/$STDOUT_NAME > /dev/null); then | ||
echo updating $MYDIR/$STDOUT_NAME | ||
cp $BUILD_DIR/$STDOUT_NAME $MYDIR/$STDOUT_NAME | ||
fi | ||
if [ -f $BUILD_DIR/$STDERR_NAME ] && \ | ||
! (diff $BUILD_DIR/$STDERR_NAME $MYDIR/$STDERR_NAME > /dev/null); then | ||
echo updating $MYDIR/$STDERR_NAME | ||
cp $BUILD_DIR/$STDERR_NAME $MYDIR/$STDERR_NAME | ||
fi | ||
done | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.