Automatically track expected tokens while parsing

[casey]

Remove all manual tracking of which tokens would have been accepted by
the parser in favor of having the parser add tokens that it checks for
to a set of expected tokens, clearing them when it accepts a token, and
using the current contents of the set in error messages.

This is a massive improvement, and will make the parser easier to
modify going forward.

And, this actually solves my sole issue with hand-written parsers.

Thanks to matklad on reddit for suggesting this!

[casey]

[lunacookies]

Here’s the reddit comment in question for posterity:

I know almost nothing about producing error messages with hand written parsers: in my experience, displaying wavy underline for syntax errors synchronously with typing completely dominates actual error messages.
That said, I think usually this is handled by maintaining a runtime set of expected tokens. Each time parser consumes a token, the set is cleared. Each time parser tries to consume token foo, but sees bar, foo is added to the set. When the parser is stuck (it tried all alternatives, or there’s only a single alternative which must match), the parser prints the set into the error message. — u/matklad (https://www.reddit.com/r/rust/comments/jh69jx/blog_post_introducing_ungrammar/ga1gms8?utm_source=share&utm_medium=web2x&context=3)