Add a LALR grammar for Rust with testing support

[bleibig]

This adds a new lexer/parser combo for the entire Rust language can be generated with with flex and bison, taken from my project at https://github.com/bleibig/rust-grammar. There is also a testing script that runs the generated parser with all *.rs files in the repository (except for tests in compile-fail or ones that marked as "ignore-test" or "ignore-lexer-test"). If you have flex and bison installed, you can run these tests using the new "check-grammar" make target.

This does not depend on or interact with the existing testing code in the grammar, which only provides and tests a lexer specification.

OS X users should take note that the version of bison that comes with the Xcode toolchain (2.3) is too old to work with this grammar, they need to download and install version 3.0 or later.

The parser builds up an S-expression-based AST, which can be displayed by giving the "-v" argument to parser-lalr (normally it only gives output on error). It is only a rough approximation of what is parsed and doesn't capture every detail and nuance of the program.

Hopefully this should be sufficient for issue #2234, or at least a good starting point.

[rust-highfive]

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

[steveklabnik]

😍

[emberian]

+10000, so glad to finally see this work start to move in-tree.

[erickt]

Wow, this is awesome. Did any of these tools find any ambiguity in the grammar?

[hawkw]

+1

[aturon]

Whoa...!

[zwarich]

Strictly speaking, this isn't a LALR grammar for Rust, because it relies on the use of a push_back function in semantic actions to push tokens back onto the token stream.

[brson]

This looks pretty awesome. What are the prospects for testing that files the Rust parser rejects are also rejected by the grammar?

[jbclements]

I love where this is headed!

[bleibig]

The grammar as it stands has a number of S/R conflicts, but they are all resolved through use of the precedence features in bison to (hopefully) match how the production rust parser works in these situations.

As far as testing goes, the testing script does not do negative tests for programs that should fail to parse, but that feature can easily be added to the script. We can do that using programs in the compile-fail directory, however not all files there fail to parse as they are meant to fail in a later stage of compilation. We can check whether it's supposed to parse first with rustc -Z parse-only, but it might be a better idea to split the compile-fail directory so that files that fail to parse are in a new "parse-fail" directory.

[nikomatsakis]

@bors r+ f39297f

[bors]

⌛ Testing commit f39297f with merge 4e4e8cf...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-x-android-t, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-win-32-nopt-t, auto-win-32-opt, auto-win-64-nopt-t, auto-win-64-opt

[sanxiyn]

Where should one send changes to grammar now? bleibig/rust-grammar or rust-lang/rust?

[steveklabnik]

Rust-lang/rust

[keleshev]

This grammar is likely not LALR(1). When an ambiguity exists in an LALR or LR grammar, it could be resolved in two ways, either by:

1. rewriting the grammar to resolve ambiguity, or by
2. employing heuristics, for example, deciding whether to shift or reduce based on operator precedence.

In first case, the grammar is guaranteed to be (LA)LR, but in second—it might or might not. This Rust grammar is resolving ambiguities with precedence, so—it might not be (LA)LR.

Example

expr:
 | expr "+" expr
 | expr "*" expr
 | NUBMER

It is ambiguous because 1 + 2 * 3 could be parsed into either
(1 + 2) * 3 or 1 + (2 * 3), so it is not (LA)LR.

To resolve the ambiguity, you can rewrite this grammar as:

expr:
 | NUMBER 
 | sum

sum:
 | product "+" product

product:
 | expr "*" expr

Now, it is LALR(1) and it will parse the expression only as 1 + (2 * 3).

But, practically speaking, most (LA)LR parser generators allow you to resolve grammar ambiguities with precedence, so this is probably not a big deal.

[keleshev]

This grammar seems to define assignment and compound assignment operators as left-associative (which is corresponds to reference description), however this example confirms that = is right-associative:

fn main() {
  let mut u: ();
  let mut a: u8;

  u = (a = 2);  // right associativity
//(u = a) = 2;     left associativity, doesn't work
  u = a = 2;    // this works, so it must be right-associative

  print!("{} {}", a, u == ());
}

[keleshev]

Here is the place where erroneous left-associativity of assignment operators is defined.