Skip to content

Commit

Permalink
Fix incorrect keyword lexing
Browse files Browse the repository at this point in the history
For some weird reason this line is detected as a keyword line:

```
            type: :module,
```

Though it clearly isn't. Ripper:

```
require 'ripper'

pp Ripper.lex(<<~'EOM')
  {
    type: :module,
  }
EOM
```

Produces:

```
[[[1, 0], :on_lbrace, "{", BEG|LABEL],
 [[1, 1], :on_ignored_nl, "\n", BEG|LABEL],
 [[2, 0], :on_sp, "  ", BEG|LABEL],
 [[2, 2], :on_label, "type:", ARG|LABELED],
 [[2, 7], :on_sp, " ", ARG|LABELED],
 [[2, 8], :on_symbeg, ":", FNAME],
 [[2, 9], :on_kw, "module", ENDFN],
 [[2, 15], :on_comma, ",", BEG|LABEL],
 [[2, 16], :on_ignored_nl, "\n", BEG|LABEL],
 [[3, 0], :on_rbrace, "}", END],
 [[3, 1], :on_nl, "\n", BEG],
 [[4, 0], :on_const, "EOM", CMDARG]]
```

This is the problem line:

```
 [[2, 9], :on_kw, "module", ENDFN],
```

Digging into the IRB source code they handled this case here ruby/ruby@776759e. Based on the description I believe this may be a bug in the lexer, but I'm not sure how to validate it.
  • Loading branch information
schneems committed Jan 4, 2022
1 parent fdf862b commit 6cc8bb0
Show file tree
Hide file tree
Showing 5 changed files with 30 additions and 5 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
## HEAD (unreleased)

## 3.1.1

- Fix case where Ripper lexing identified incorrect code as a keyword (https://github.com/zombocom/dead_end/pull/122)

## 3.1.0

- Add support for Ruby 3.1 by updating `require_relative` logic (https://github.com/zombocom/dead_end/pull/120)
Expand Down
5 changes: 4 additions & 1 deletion lib/dead_end/lex_all.rb
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@ def initialize(source:, source_lines: nil)
lineno = @lex.last.pos.first + 1
end

@lex.map! { |elem| LexValue.new(elem.pos.first, elem.event, elem.tok, elem.state) }
last_lex = nil
@lex.map! { |elem|
last_lex = LexValue.new(elem.pos.first, elem.event, elem.tok, elem.state, last_lex)
}
end

def to_a
Expand Down
12 changes: 9 additions & 3 deletions lib/dead_end/lex_value.rb
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,21 @@ module DeadEnd
class LexValue
attr_reader :line, :type, :token, :state

def initialize(line, type, token, state)
def initialize(line, type, token, state, last_lex = nil)
@line = line
@type = type
@token = token
@state = state

set_kw_end
set_kw_end(last_lex)
end

private def set_kw_end
private def set_kw_end(last_lex)
@is_end = false
@is_kw = false
return if type != :on_kw
#
return if last_lex && last_lex.fname? # https://github.com/ruby/ruby/commit/776759e300e4659bb7468e2b97c8c2d4359a2953

case token
when "if", "unless", "while", "until"
Expand All @@ -41,6 +43,10 @@ def initialize(line, type, token, state)
end
end

def fname?
state.allbits?(Ripper::EXPR_FNAME)
end

def ignore_newline?
type == :on_ignored_nl
end
Expand Down
2 changes: 1 addition & 1 deletion lib/dead_end/version.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# frozen_string_literal: true

module DeadEnd
VERSION = "3.1.0"
VERSION = "3.1.1"
end
12 changes: 12 additions & 0 deletions spec/unit/code_line_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,18 @@

module DeadEnd
RSpec.describe CodeLine do
it "bug in keyword detection" do
lines = CodeLine.from_source(<<~'EOM')
def to_json(*opts)
{
type: :module,
}.to_json(*opts)
end
EOM
expect(lines.count(&:is_kw?)).to eq(1)
expect(lines.count(&:is_end?)).to eq(1)
end

it "supports endless method definitions" do
skip("Unsupported ruby version") unless Gem::Version.new(RUBY_VERSION) >= Gem::Version.new("3")

Expand Down

0 comments on commit 6cc8bb0

Please sign in to comment.