Announcing v0.10

[PoiScript]

Hello everyone. After leaving this crate for almost unmaintained for over three years, I finally had some time to pick up what we left off. :)

Three years has passed, and many new feature and awesome crates have been introduced in the Rust world. So I think it's time for us to rebuild orgize with these new features and crates in mind. So I'm thrilled to announce that i'll be publishing orgize v0.10 in the next couple of weeks.

This version is total rewrite of orgize. Some notable changes including:

Switching to rowan

In this version, we're replacing the underlying data structure from indextree to rowan, one of the core crates of rust-analyzer.

To visualise that, input * Heading *bold* will now becomes something like:

DOCUMENT@0..17
  HEADLINE@0..17
    HEADLINE_STARS@0..1 "*"
    WHITESPACE@1..2 " "
    HEADLINE_TITLE@2..16
      TEXT@2..10 "Heading "
      BOLD@10..16
        STAR@10..11 "*"
        TEXT@11..15 "bold"
        STAR@15..16 "*"
    NEW_LINE@16..17 "\n"

rowan help us resolves some long exist problem like high memory usage and not able to get position of each parsed element.

Also thanks to the rich ecosystem of rowan, we might have a lsp server for org mode in future, just like rust-analyzer does for rust language.

Traversing org element tree

Another big change in this version is that org.iter() is now superseded by org.traverse(). When walking through org element tree, traversal is much more flexible than simple iteration. For example, you can now skip the whole subtree but continue on its sibling without breaking the entire traversing by calling traversal_context.skip().

Lossless parsing

Orgize is now a loessless parser. It means you can always get original input back from Org struct:

let s = "* hello /world/!";
let org = Org::parse(s);
assert_eq!(s, org.to_org());

Try it out today

You can try out v0.10 by installing it from crates.io:

cargo add orgize@0.10.0-alpha.1 

or using new demo website: https://poiscript.github.io/orgize

I hope you guys enjoy the new release, and feel free to send any feedback!

What's next

Before publishing v0.10, I want to fix other issues and merge pull requests as much as possible.

After that, I want to support parsing org entities and inlinetasks and try build a lsp server for org-mode using tower-lsp and orgize.

[PoiScript]

I had closed some outdated issues and pull requests, and what remains is mostly about adding new feature:

support repeater/delay in timestamp:

1. FR: Timestamps: Repeater/delay may be specified (and different) for both parts of a range #29
2. Add repeater and delay to timestamp parsing #32
3. Unable to access Timestamps for Repeating Tasks #57

support latex environment:

1. feat: parse Latex environment #47
2. Handle LaTeX fragments and environments — $x!$, $$y$$, \begin{equation}z\end{equation} #42

support org entities:

1. Support for org-entities (special symbols) #44

and bug about handling special type of whitespace:

1. Orgize validation fails when parsing certain unicode values #22
2. \r\n prevents further parsing; potential for substantial data loss #26
3. Add a simple unicode fuzz test as a standalone example. #23

[justinabrahms]

Hey PoiScript. Thanks for the update. I've forked orgize to add a few things I needed, so I'd love to get them into 0.10. You've fixed the exporter need I have w/ the Traverser trait. The one thing I'm missing now is that I need access to an iterable of ancestors when evaluating each node.

You can see my expected use-case here: https://github.com/justinabrahms/org-roam-to-subtext/blob/main/src/main.rs#L65-L72

I need to know if I'm within a quote so I can prefix Text lines with > for the export format I'm targeting, because it's line based not block based.

This was my commit which added it to the older code base: 8f66d5c

[PoiScript]

hi @justinabrahms, it's possible to access the ancestors of parsed element by using some low-level apis in rowan:

use orgize::SyntaxKind;

fn text(&mut self, token: SyntaxToken, _ctx: &mut TraversalContext) {
    let inside_quote = token
        .parent_ancestors()
        .any(|n| n.kind() == SyntaxKind::QUOTE_BLOCK);

    // ... 
}

but it's not performant for your use case, since you have to check ancestors in each text. I would recommend you to store a flag in your traverser, and set it in quote_block:

pub struct MyTraverser {
    inside_quote: bool
}

fn text(&mut self, token: SyntaxToken, _ctx: &mut TraversalContext) {
    self.inside_quote
    // ... 
}

fn quote_block(&mut self, event: WalkEvent<&QuoteBlock>, _ctx: &mut TraversalContext) {
    self.inside_quote = matches!(event, WalkEvent::Enter(_));
}

[justinabrahms]

So I've spent a bit of time this morning attempting to convert my tool to v0.10. Some observations:

1. We need a plaintext handler to implement b/c overriding all of the htmlisms is going to be very tedious when implementing a plain-text format. Defaulting to org syntax seems fine to me.
2. I can't actually fetch the underlying data in, for instance, a Keyword.. because all of the methods to actually get at the syntax are private or crate-private. This means I can't, for instance, get the #+title: document element. I'd expect to be able to get the key and the value separately.
3. The forward macro errors if you don't import orgize::rowan::WalkEvent which was surprising. Not sure how common that is in rust macro land.
4. I have some minor concerns about what might happen to the stability of the API if rowan either falls out of favor or gets a backwards incompatible change, given we're straight re-exporting their stuff. You know more about the library and their compatibility guarantees, so I'll trust you on it.
5. finish() as a "gimmie my string please" indicates finality, but you could call it multiple times without error. I would have either expected something like render() or to just implement a string conversion.

[PoiScript]

hi, @justinabrahms, thanks for the feedback!

I also find out it's tedious to impl Traverser trait, since it requires too many methods to be defined. So I've combined all these methods in a single event method in next release.

And now it should be pretty straightforward to create a plain text format:

struct PlainText(String);

impl Traverser for PlainText {
    fn event(&mut self, event: Event, ctx: &mut TraversalContext) {
        if let Event::Text(text) = event {
            self.0 += text.text();
        }
    }
}

I have some minor concerns about what might happen to the stability of the API if rowan either falls out of favor or gets a backwards incompatible change, given we're straight re-exporting their stuff.

We re-export rowan crate for convenience. User should only call its api for advanced usage. So it'll be fine as long as rowan follows semantic versioning. If you do reply on rowan api and are worried about breaking changes, you can pin its in Cargo.toml.

finish() as a "gimmie my string please" indicates finality, but you could call it multiple times without error.

You cannot call finish more than one time, because it consumes self.

[justinabrahms]

@PoiScript I've picked this back up and have tried to get it working, but I can't get it to 100%. The parsing changes do feel better, so thank you.

I'm unclear how to print out the AST of what the system thinks my document structure is. I need this for debugging. The main problem is all of the SyntaxNode attributes are hidden from me, so I can't dump them via serde or similar. Rowan appears to have support for this. Can we get a method for this on Document?

In a similar vein, you've removed the ability for me to look at the ancestry of my node b/c I no longer get a SyntaxToken in the event, so I've had to adopt various boolean flags, which I don't prefer.

The API for Headline feels a bit odd. I'd have expect a method there to just return a text string, rather than the iterator that's there now. I'm guessing org is weirder than I expect here?

[PoiScript]

@justinabrahms

I'm unclear how to print out the AST of what the system thinks my document structure is.

You can import the orgize::rown::AstNode trait to do so:

use orgize::Org;
use rowan::ast::AstNode;

fn main() {
    let org = Org::parse("* hello");
    println!("{:#?}", org.document().syntax());
}

AstNode is implemented for all elements.

Also, you use the demo website: https://poiscript.github.io/orgize/, the structure of you input will be shown under the 'Syntax' tab

b/c I no longer get a SyntaxToken in the event

I don't really get it, did you mean the Event::Text variant?

The API for Headline feels a bit odd. I'd have expect a method there to just return a text string, rather than the iterator that's there now.

Headline title can contain other elements, like bold and italic. So Headline::title() actually returns "parsed" title. I just added a new method Headline::title_raw() for returning the title raw string.

[justinabrahms]

@justinabrahms

I'm unclear how to print out the AST of what the system thinks my document structure is.

You can import the orgize::rown::AstNode trait to do so:

use orgize::Org;
use rowan::ast::AstNode;

fn main() {
    let org = Org::parse("* hello");
    println!("{:#?}", org.document().syntax());
}

AstNode is implemented for all elements.

Wild. I've never seen an import change the behavior like that, though I'm a rust newbie. This is the error I was getting otherwise. Do you know the search term for what this pattern is called so I can learn more?

error[E0599]: no method named `syntax` found for struct `orgize::ast::Document` in the current scope
   --> src/main.rs:195:39
    |
195 |     println!("{:#?}", tree.document().syntax());
    |                                       ^^^^^^ private field, not a method
    |
   ::: /home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rowan-0.15.15/src/ast.rs:40:8
    |
40  |     fn syntax(&self) -> &SyntaxNode<Self::Language>;
    |        ------ the method is available for `orgize::ast::Document` here

b/c I no longer get a SyntaxToken in the event

I don't really get it, did you mean the Event::Text variant?

I'm good now. I don't think I realized that the text element was being passed a rowan::SyntaxToken.

The API for Headline feels a bit odd. I'd have expect a method there to just return a text string, rather than the iterator that's there now.

Headline title can contain other elements, like bold and italic. So Headline::title() actually returns "parsed" title. I just added a new method Headline::title_raw() for returning the title raw string.

I don't think you need to add this. I forgot that headlines in org were rich so this makes sense now.

<3

I'm a happy camper now. Thanks for your patience.

[rynoV]

Do you know the search term for what this pattern is called so I can learn more?

The pattern is just rust's traits, in this case there is a trait called AstNode defined in rowan which provides the syntax method. I also found it unintuitive while getting into rust since there's no obvious connection between what you have to import and what you're trying to use. I guess it also didn't help that orgize's Document had a private field with the same name, which made the error a bit more confusing than it normally would have been, but I think the second part of the error you showed normally helps to figure out what to import in this scenario.

[PoiScript]

thank @rynoV for explaining! I actually don't have such problem when writing rust code, since rust-analyzer should be smart enough for locating and importing trait you want:

[gmemstr]

I find the removal of the .keywords() functionality a little frustrating, and haven't found a satisfactory replacement for it - my use case is a static site generator, and my posts typically have #+title and #+date defined for displaying and also organization. Is this function planned to be re-implemented, or is it something we'll have to rely on the traversal functionality in our own code to access?

[rynoV]

@gmemstr I'm not sure exactly what keywords did but I think this should do what you need:

use orgize::{Org, ast::Keyword};
use rowan::ast::{support, AstNode};

let org = Org::parse("#+KEY1: VALUE1\n#+KEY2: VALUE2\nabc");
let keywords: Vec<Keyword> =
support::children::<Keyword>(org.document().section().expect("No section").syntax()).collect();

assert_eq!(keywords.len(), 2);

assert_eq!(keywords[0].key(), "KEY1");
assert_eq!(keywords[0].value(), " VALUE1");

assert_eq!(keywords[1].key(), "KEY2");
assert_eq!(keywords[1].value(), " VALUE2");