Skip to content

woxtu/rust-tinysegmenter

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

rust-tinysegmenter

Crates.io

Rust implementation of TinySegmenter, which is a compact Japanese tokenizer.

Install

Adding the following to the Cargo.toml in your project:

[dependencies]
tinysegmenter = "0.1"

and import using extern crate:

extern crate tinysegmenter;

Usage

let tokens = tinysegmenter::tokenize("私の名前は中野です");
println!("{}", &tokens.join("|")); // 私|の|名前|は|中野|です

License

Copyright (c) 2015 woxtu

Licensed under the MIT license.

About

Compact Japanese tokenizer

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages