Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[native] Add a rust implementation of whitespace_parser #452

Closed
wants to merge 1 commit into from
Closed

[native] Add a rust implementation of whitespace_parser #452

wants to merge 1 commit into from

Commits on Jan 24, 2021

  1. [native] Add a rust implementation of whitespace_parser 

    This adds an experimental new native extension, with the goal of
improving performance, currently just reimplementing
`whitespace_parser`.

This appears to make parsing marginally faster overall, but the
performance gains are pretty limited because whitespace parsing is a
small portion of the overall parsing process, and there's some added
overhead of converting types (mostly unicode strings) between rust and
python.

Instead, this helps show what's needed to port part of the codebase over
to rust in an example that's small enough to be digestible, but big
enough to be more than a trivial toy.

I originally started by trying to port the tokenizer to rust, since it
takes a larger portion of time and it's infrequently modified (making it
a good candidate), but found issues with the implementation direction I
was taking, so I scrapped that for now.
    @bgw
    bgw committed Jan 24, 2021
    Configuration menu
    Copy the full SHA
    10b89be View commit details
    Browse the repository at this point in the history