[Lexer] Allow multiline delimiter to be escaped in raw Strings

[maustinstar]

The Lexer currently eats triple-quotes as a multiline delimiter for Strings. This Pull request allows the delimiter to be escaped in raw Strings when the correct number of terminating '#' marks are found at the end of the line.
E.g.
#"""# is now equivalent to "\""
#"""text"# equates to "\"\"text"
#""" is still the beginning of a multiline raw string.

Resolves SR-10011.

[theblixguy]

cc @rintaro

[rintaro]

Hi Michael, welcome to the project!
Thank you for the PR. This is a good start!

For testing, the content of the literals are tested in test/Parse/raw_string.swift. Please add test cases there. As for error cases like #"""meow#\n, please add them to test/Parse/raw_string_errors.swift.

[maustinstar]

Tests have been moved; logic has been rewritten. Thank you for the warm welcome:)

[xwu]

This is awesome! Minor nits :)

[rintaro]

Implementation looks good! Could you add several test cases?

[rintaro]

@swift-ci Please smoke test

[rintaro]

@swift-ci Please smoke test

[rintaro]

******************** TEST 'Swift(linux-x86_64) :: Parse/raw_string_errors.swift' FAILED ********************
Script:
--
: 'RUN: at line 1';   /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/swift-linux-x86_64/bin/swift -frontend -target x86_64-unknown-linux-gnu  -module-cache-path '/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/buildbot_linux/swift-linux-x86_64/swift-test-results/x86_64-unknown-linux-gnu/clang-module-cache' -swift-version 4   -typecheck -verify -disable-objc-attr-requires-foundation-module /home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/swift/test/Parse/raw_string_errors.swift
--
Exit Code: 1

Command Output (stderr):
--
/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/swift/test/Parse/raw_string_errors.swift:15:15: error: unexpected error produced: multi-line string literal content must begin on a new line
let _ = ###"""invalid"##
              ^
/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/swift/test/Parse/raw_string_errors.swift:19:4: error: expected error not produced
// expected-error@-1{{too many '#' characters in closing delimiter}}{{24-27=}}
~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/swift/test/Parse/raw_string_errors.swift:20:4: error: expected error not produced
// expected-error@-2{{consecutive statements on a line must be separated by ';'}}
~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/swift/test/Parse/raw_string_errors.swift:21:4: error: expected error not produced
// expected-error@-3{{expected expression}}
~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/swift/test/Parse/raw_string_errors.swift:26:4: error: expected error not produced
// expected-error@-1{{multi-line string literal content must begin on a new line}}{{14-14=\n}}
~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/swift/test/Parse/raw_string_errors.swift:27:4: error: expected error not produced
// expected-error@-2{{multi-line string literal closing delimiter must begin on a new line}}{{5-5=\n}}
~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/buildnode/jenkins/workspace/swift-PR-Linux-smoke-test/branch-master/swift/test/Parse/raw_string_errors.swift:31:4: error: expected error not produced
// expected-error@-1{{multi-line string literal content must begin on a new line}}{{14-14=\n}}
~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

--

********************

This is because let _ = ###"""invalid"## eats the rest of the file. Feel free to split this test case to another file.

[maustinstar]

Changed the test case to something that won't eat the whole file but still ensures that the raw string would act like a single-liner.

Sorry for being out of touch for so long.

[rintaro]

@swift-ci Please smoke test

[maustinstar]

Does delimiterMatches return false when the lhs is shorter than the rhs? (e.g. ##"string"###). If so, then tests like the ones I wrote won't work because the logic I wrote will only treat the raw string as a one-liner if the delimiterMatches is true before a line break. My guess is that delimiterMatches is not returning true when rhs is longer than lhs.

Should I change the logic to look for any delimiter (not necessarily of matching length) and treat the raw string as a one-liner? And inherently the parse tests would behave like the one-line counterparts?

[shahmishal]

Please update the base branch to main by Oct 5th otherwise the pull request will be closed automatically.

• How to change the base branch: (Link)
• More detail about the branch update: (Link)

[xwu]

@maustinstar Are you still interested in working on this issue?

[maustinstar]

@xwu If someone else wants to take it, that's okay. But if I attempt this again, I'll start from scratch and open a different PR.