Skip to content
This repository has been archived by the owner on Jan 25, 2022. It is now read-only.

Add unicode escapes, allow non-ASCII identifiers & others improvements #136

Open
wants to merge 11 commits into
base: master
Choose a base branch
from

Conversation

nibags
Copy link

@nibags nibags commented Jun 25, 2018

Changes:

  • Fix escape characters. The detection of escapes is very general, I have replaced it with the specific escapes supported by Rust [1].
  • Add unicode escapes: \u{NNNNNN} [2]. However, the bytes and byte strings don't allow unicode escapes [3], therefore, the unicode escape is placed in a different object and new objects are created for byte strings and byte characters.
  • Fix: remove obsolete integer suffixes is & us (these were depreciated by isize & usize [4]).
  • Allow non-ASCII characters in identifiers (in the Rust parser, an identifier is: [a-zA-Z\x80-\xff_][a-zA-Z0-9\x80-\xff_]* [5]). This affects Lifetimes, Macros, functions and types (to do this, I basically added \\x80-\\xFF, and it works fine. But the POSIX character class [:ascii:] can also be used, for example: (?:[a-zA-Z0-9_]|[^[:ascii:]]) ).
  • Add missing sigils [6]: ~ @

NOTE: The use of non-ASCII characters in identifiers is a feature of Rust, however these have some issues and aren't currently fully supported (there is even the possibility that it will be eliminated in the future) (see ref. [7]).

If you want to make some change, correction, or undo something, I have no problem.

References:

Some Sources:

@zargony
Copy link
Owner

zargony commented Dec 10, 2018

Thanks for submitting this changes! And sorry for the late response...

Unicode escapes look good to me. It would be nice to also have it covered by tests.

Are ~ and @ sigils are still used? I though they were deprecated a long time ago?

@nibags
Copy link
Author

nibags commented Dec 11, 2018

Yes, @ and ~ are deprecated! I think I better remove them.
I also added the tests

@nibags
Copy link
Author

nibags commented Dec 11, 2018

Sorry, the test I did was making mistakes and I made unnecessary commits

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants