[close #64 #76] Refactor CodeLine add CleanDocument "sweep" class

[schneems]

Refactor CodeLine

Previously CodeLine knew how to "lex" itself. This strategy worked for many cases but failed pretty hard in cases where context can make the same line return drastically different results.

For example, by itself this line:

EOM

Will be "lex"-d into a constant while more lines:

foo = <<~EOM
  Hello world
EOM

It would be understood as a heredoc.

Distinct "clean" phase added via CleanDocument

Even after we've "injected" the lexical information into CodeLine based on lexing the entire document, there are still problems with special cases such as #64, where multiple lines represent a single logical operation. Heredocs are also painful to work with.

Previously, a HeredocBlockParse would put all heredocs into a CodeBlock at the beginning of the CodeSearch. There was also a TrailingSlashJoin and a method for removing code comments.

All of that functionality has been moved to a single class, CleanDocument, responsible for prepping the source to be in a good state to be fed into CodeSearch.

Join consecutive logical blocks in CleanDocument

To close #64, the CleanDocument can use information in the lex line to join multiple "consecutive" lines like:

User.
  where(name: 'schneems')

This code style is a problem because the second line looks like valid ruby code in isolation, but if it's removed, an error might be introduced (due to the trailing dot on User.)

More notes around this here https://gist.github.com/schneems/6a7d7f988d3329fb3bd4b5be3e2efc0c and #76.

Essentially it uses lex information to join into a single line:

"User.\nwhere(name: 'schneems')\n"

Once in a single line, the search algorithm cannot accidentally introduce an error into the document by removing part of it.

CaptureCodeContext

This class is now being billed as a "third" phase for re-introducing ambiguity where it should logically exist. Previously we were handling the case where a keyword was missing an end at the end of a block like:

class Dog
  def bark
end

It was also determined there's a logical inverse of this that wasn't being covered:

class Dog
  end
end

It's now handled.

For the capture_before_after_kws method, it was determined that it only needed to remove ambiguity if there's only one visible line. Adding this check cleaned up several outputs and didn't remove any critical (or ambiguous) lines.

Other

• Add a bunch of docs all over the place
• Add deprecation to requiring dead_end/fyi
• Move LexValue class to its own file