Error: Indexing into an empty node

[Marwes]

Attempt to skip over /* */ comments but I am getting an error which seems to be a bug. Similar to #97 I think as I were getting that error before reducing it.

#[derive(logos::Logos)]
pub enum Token {
    #[error]
    #[regex(r"/\*([^\*]*\*+[^\*/])*([^\*]*\*+|[^\*])*\*/", logos::skip)]
    Error,
}

[Marwes]

#[derive(logos::Logos)]
pub enum Token2 {
    #[regex(r#"[!#$%&*+-./<=>?@\\^|~:]+"#)]
    Operator,

    #[error]
    #[regex(r"/\*([^\*]*\*+[^\*/])*([^\*]*\*+|[^\*])*\*/", logos::skip)]
    Error,
}

Gives Merging two reserved nodes!

[Evrey]

Your regex is very strange.

(?x:
    # (1) This entire block…
    (   [^\*]*
        \*+
        [^\*/]
    )*
    # (1) … is almost the exact same as this block. And both are optional.
    (   [^\*]*
        \*+     # (2) Why `+ |` when you can do `*`?
    |   [^\*]
    )*
)

I think what you wanted was this? (?x: [^\*] | (\* [^/]?) ).

And in your Token2 example, your regexes clash. Logos sadly can't roll back to commit multiple tokens. So when you were to write a /* b, is that the start of a block comment now, or is it a custom operator? Logos needs to find the final */ to decide. Now imagine the full code looks like this: a /* b + c */. If the final */ didn't exist, Logos would have to go back and commit three identifier and two operator tokens.

So the »bug« for Token2 is that the error message should say: »Thine token rules are too powerful for me.«

You can solve this in two ways, an elegant and a hack-y way:

• The elegant solution is to not have comment delimiters overlap with other syntax elements. This is of course not an option if you are parsing an already existing proglang. Edit: For example, comments start with \#\x20 for me. No custom operator can have # anywhere inside.
• The hack-y solution is to make a #[token("/*")] MaybeBlockCommentStart, and then branch off to a sub-lexer via Lexer::morph and let it either find the end of a block comment or revert all sub-lexing and commit an operator. This, of course, is costly infinite look-ahead.
• Edit: Well, there's a third way I forgot about. Explicitly disallow /* as an operator. For example, in my own lexer, : is a special token, so only :: may appear in an operator. ::= is a valid custom operator, := is not.

[maciejhirsz]

I should have figured out this is going to be a problem. There was an issue before when the first thing inside a grouped repeat was a maybe: (f?oo)*. I fixed that one, but haven't thought about the same issue with loops that begin with a loop: (f*oo)*. I should have a fix shortly.

[Marwes]

I think I may have modified the regex from this answer https://stackoverflow.com/a/36328890/2489366 so that it didn't need non-greedy matching since that caused ambiguities with LALRPOP. Then I just copied that into logos when adding comments. There may very well be a better way to write it in logos.

The error is still poor, as noted so I didn't actually know what I were supposed to do to fix it :)

[maciejhirsz]

The error is still poor, as noted so I didn't actually know what I were supposed to do to fix it :)

Any panic in logos-derive is a bug at this point, I'm fixing it :)

[maciejhirsz]

I should have a fix shortly.

Famous last words. So I fixed the issue I thought this was relatively quickly, but the example you gave me will require a bit more digging.

The operator and comment regexes overlap, and they are both looping. I've handled this case elsewhere quite gracefully, but Logos really doesn't like this example. That it panics is bad enough, but it panics after a long time, which means there is some crazy looping going on when merging the regexes.

I'll make a release with the fix I have and #131, and get back to tracking this.

Edit: shortly = almost two days ;P

[maciejhirsz]

Released a fix in 0.11.3 (logos-derive 0.11.5)