Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transliterator runtime #3946

Merged
merged 95 commits into from
Aug 31, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
e0dcb85
wip
skius Aug 24, 2023
8c79370
wip
skius Aug 25, 2023
0a86ca8
wip
skius Aug 25, 2023
59ddc1f
Merge branch 'main' into tl-runtime
skius Aug 25, 2023
9976099
wip
skius Aug 25, 2023
be4359b
fmt
skius Aug 25, 2023
60ce069
wip
skius Aug 25, 2023
cc8532a
simple things are working :)
skius Aug 25, 2023
8b52bc2
add test transform rules
skius Aug 25, 2023
7713ec6
test in bakeddata
skius Aug 25, 2023
15397da
more
skius Aug 25, 2023
e11279a
wip
skius Aug 25, 2023
6e8eed1
switch to input str wrapper for anchor support
skius Aug 25, 2023
5097871
support segments + backrefs
skius Aug 25, 2023
0006bb9
quantifiers, but broken because of dynamic segment numbering
skius Aug 26, 2023
2419524
Squash of segments with indices
skius Aug 26, 2023
2e80fa9
make quantifiers work (including with segments)
skius Aug 26, 2023
d384afc
add function call + Any-Remove test
skius Aug 26, 2023
57b2e1a
setup for function call replacement
skius Aug 26, 2023
9f29e9c
add ignore_len to replaceable and therefore make function calls work
skius Aug 26, 2023
7bafbe6
fmt
skius Aug 26, 2023
9abee53
useless assert
skius Aug 26, 2023
d820a5c
regenerate testdata
skius Aug 26, 2023
5d8f97e
Merge branch 'main' into tl-runtime
skius Aug 26, 2023
9fcfffe
fix cursor placeholder filter interaction
skius Aug 26, 2023
2b90c2c
fix rule replacement application
skius Aug 26, 2023
b289160
fix run generation
skius Aug 26, 2023
035cb9f
add empty match test
skius Aug 26, 2023
2809374
fmt
skius Aug 26, 2023
9ff0945
typo
skius Aug 26, 2023
3f818f5
switch from raw_cursor to visible_content-relative cursor
skius Aug 26, 2023
70275a3
refactor set_ignore_len into constructor
skius Aug 26, 2023
3b60b3f
no unnecessary allocations
skius Aug 26, 2023
7e7cb4d
fix safety violations with cursor offset
skius Aug 26, 2023
7a14dd6
add replacement size hints
skius Aug 26, 2023
67d0775
reuse work when checking for encoded chars
skius Aug 26, 2023
66be38b
add custom transliterator support
skius Aug 26, 2023
437443b
add notes
skius Aug 26, 2023
2b3d91e
make normalization tests pass, but baked is probably wrong
skius Aug 26, 2023
2c12542
use other crates' baked providers
skius Aug 26, 2023
fe5ed45
add cldr testData tests for supported locales
skius Aug 27, 2023
a270ce5
300k lines is maybe a bit much
skius Aug 27, 2023
41b1fd0
add notes
skius Aug 27, 2023
a03f2f8
add convenience function
skius Aug 27, 2023
2255447
cleanup
skius Aug 27, 2023
d1f880d
cleanup
skius Aug 27, 2023
3ccc422
cursoroffset::byte safety
skius Aug 27, 2023
283ba60
notes
skius Aug 27, 2023
cd7091d
switch to safer Matcher API
skius Aug 27, 2023
5e0e3b2
refactor RepMatcher to avoid some index recomputations
skius Aug 27, 2023
7d1fb9e
benchmarking
skius Aug 27, 2023
5370bc2
remove eprintlns
skius Aug 27, 2023
ba16b35
enforce no key matching after post matching
skius Aug 27, 2023
88d037d
avoid potential cursor safety issue
skius Aug 27, 2023
c88e2f4
remove duplicate license header
skius Aug 27, 2023
7822602
improve perf by '25%'
skius Aug 27, 2023
0ef369f
fix backref bug
skius Aug 27, 2023
3932436
safe replaceable-from-insertable getting
skius Aug 28, 2023
c33bc8d
replaceable api changes
skius Aug 28, 2023
c5f521b
factor out partially-invisible-vec semantics into Hide
skius Aug 28, 2023
74e1e8f
doc changes
skius Aug 28, 2023
cf82d85
remove last unsafe in transliterator module
skius Aug 28, 2023
f942dfb
more docs for replaceable.rs
skius Aug 28, 2023
5ba561d
Merge branch 'main' into tl-runtime
skius Aug 28, 2023
6286c28
regenerate testdata
skius Aug 28, 2023
8dff4d9
more comments
skius Aug 28, 2023
7e08042
trait docs
skius Aug 28, 2023
595fd38
more notes
skius Aug 28, 2023
738b19c
fmt
skius Aug 28, 2023
3f293de
doc additions
skius Aug 29, 2023
85b9f23
docs
skius Aug 29, 2023
f7ad0a5
turn non-actionable todos into thoughts and questions
skius Aug 29, 2023
690d62b
todos
skius Aug 29, 2023
99fb3d5
todos
skius Aug 29, 2023
f607ad3
more todos
skius Aug 29, 2023
529b598
todos
skius Aug 29, 2023
feb1aeb
todos => thought
skius Aug 29, 2023
683256d
clippy lints
skius Aug 29, 2023
dd94b00
doc + std errors + fmt
skius Aug 29, 2023
1445763
license?
skius Aug 29, 2023
5c22c95
license?
skius Aug 29, 2023
97a3610
fix failing test
skius Aug 29, 2023
07be76d
fix license header
skius Aug 29, 2023
cf4839a
update lib docs
skius Aug 29, 2023
c48fcec
link to locale
skius Aug 29, 2023
2beb660
fix typo
skius Aug 29, 2023
230f3b9
typo
skius Aug 29, 2023
3745d5c
as_replaceable make_contiguous thoughts
skius Aug 30, 2023
c14389e
typo
skius Aug 30, 2023
9493143
review 1
skius Aug 31, 2023
9ab9f29
review 2
skius Aug 31, 2023
2fe0324
Merge branch 'main' into tl-runtime
skius Aug 31, 2023
25e3e41
rename in expectation of cldr#3143
robertbastian Aug 31, 2023
7345d9c
wire up benchmark
robertbastian Aug 31, 2023
79e2363
let-else
robertbastian Aug 31, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions experimental/transliteration/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@ all-features = true

[dependencies]
icu_provider = { workspace = true, features = ["macros"] }
icu_locid = { workspace = true }
icu_collections = { workspace = true }
icu_normalizer = { workspace = true }
icu_casemap = { workspace = true }

zerovec = { workspace = true, features = ["derive"] }
zerofrom = { workspace = true, features = ["derive"] }
yoke = { workspace = true, features = ["derive"] }
databake = { workspace = true, optional = true, features = ["derive"] }
litemap = { workspace = true }
Expand All @@ -43,7 +43,7 @@ criterion = "0.4.0"
default = ["compiled_data"]
std = []
serde = ["dep:serde", "zerovec/serde", "icu_collections/serde"]
datagen = ["serde", "dep:databake", "zerovec/databake", "icu_collections/databake"]
datagen = ["serde", "dep:databake", "zerovec/databake", "icu_collections/databake", "std"]
compiled_data = ["dep:icu_transliteration_data", "icu_normalizer/compiled_data", "icu_casemap/compiled_data"]

[[bench]]
robertbastian marked this conversation as resolved.
Show resolved Hide resolved
Expand Down
2 changes: 1 addition & 1 deletion experimental/transliteration/benches/bench.rs
robertbastian marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

use criterion::{black_box, BenchmarkId, Criterion};
use criterion::{criterion_group, criterion_main};
use icu_provider::_internal::locid::Locale;
use icu_locid::Locale;

use icu_transliteration::Transliterator;

Expand Down
1 change: 0 additions & 1 deletion experimental/transliteration/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@
#![warn(missing_docs)]

extern crate alloc;
extern crate core;

pub mod provider;

Expand Down
2 changes: 1 addition & 1 deletion experimental/transliteration/src/provider.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ use icu_collections::{
codepointinvlist::{CodePointInversionList, CodePointInversionListULE},
codepointinvliststringlist::CodePointInversionListAndStringListULE,
};
use icu_provider::prelude::*;
use zerovec::*;

#[cfg(feature = "compiled_data")]
Expand All @@ -33,7 +34,6 @@ const _: () = {
mod icu {
pub use crate as transliteration;
pub use icu_collections as collections;
pub use icu_normalizer as normalizer;
}
icu_transliteration_data::impl_transliterator_rules_v1!(Baked);
};
Expand Down
Loading
Loading