Transliterator DatagenProvider #3877

skius · 2023-08-16T14:46:26Z

Depends on #3872
Part of #3736

This PR

adds datagen support for transliterators
adds sample transliterator definitions to the checked in cldr-json as a workaround until they land upstream
adds compiled data support for transliterators

commit 7f84837 Author: Shane F. Carr <shane@unicode.org> Date: Thu Aug 17 19:31:59 2023 -0500 Update request.rs commit 905e697 Author: Shane F. Carr <shane@unicode.org> Date: Thu Aug 17 16:50:55 2023 -0700 gn-gen commit 1e52541 Author: Shane F. Carr <shane@unicode.org> Date: Thu Aug 17 16:50:06 2023 -0700 Update sizes (probably need to fix nightly sizes too) commit 45778ce Author: Shane F. Carr <shane@unicode.org> Date: Thu Aug 17 16:47:06 2023 -0700 Use the TinyStr variant commit 04ee1df Author: Shane F. Carr <shane@unicode.org> Date: Thu Aug 17 16:42:26 2023 -0700 Remove Default impl commit 3d83b95 Author: Shane F. Carr <shane@unicode.org> Date: Thu Aug 17 16:42:02 2023 -0700 Add impls based on Deref commit 3b14fbd Author: Shane F. Carr <shane@unicode.org> Date: Thu Aug 17 16:10:47 2023 -0700 Add 3-way enum (not using Stack variant yet) commit cda1498 Author: Shane F. Carr <shane@unicode.org> Date: Thu Aug 17 13:57:26 2023 -0700 Fix build post merge commit 71c1087 Author: Shane F. Carr <shane@unicode.org> Date: Thu Aug 17 13:56:30 2023 -0700 Support multipart auxiliary keys commit f31643a Author: Shane F. Carr <shane@unicode.org> Date: Thu Aug 17 13:17:17 2023 -0700 Use `+` for aux key commit dbefbf9 Author: Shane F. Carr <shane@unicode.org> Date: Thu Aug 17 13:04:55 2023 -0700 More more is_und commit f8e310e Author: Shane F. Carr <shane@unicode.org> Date: Thu Aug 17 13:03:07 2023 -0700 is_und in driver.rs (post merge cleanup) commit 33fad72 Merge: 5551d6b ea1ba9f Author: Shane F. Carr <shane@unicode.org> Date: Thu Aug 17 12:58:16 2023 -0700 Merge branch 'main' into auxkey Conflicts: provider/datagen/src/lib.rs commit 5551d6b Author: Shane F. Carr <shane@unicode.org> Date: Wed Aug 16 17:07:24 2023 -0700 fmt and refactor commit cd0d86c Author: Shane F. Carr <shane@unicode.org> Date: Wed Aug 16 17:00:56 2023 -0700 Fix baked_exporter.rs commit 241b563 Author: Shane F. Carr <shane@unicode.org> Date: Wed Aug 16 14:51:39 2023 -0700 Pull the separator character into a function where possible. commit 93f0969 Author: Shane F. Carr <shane@unicode.org> Date: Wed Aug 16 14:47:19 2023 -0700 Handle $ in baked_exporter commit 3493b75 Author: Shane F. Carr <shane@unicode.org> Date: Tue Aug 15 19:24:09 2023 -0700 Use is_und instead of is_empty more consistently in fallback iterator commit 52ab723 Author: Shane F. Carr <shane@unicode.org> Date: Tue Aug 15 19:19:23 2023 -0700 Add DataLocale::is_und; gen hello world testdata commit f36b405 Author: Shane F. Carr <shane@unicode.org> Date: Tue Aug 15 19:05:19 2023 -0700 Use auxiliary keys in HelloWorldProvider commit 54101ad Author: Shane F. Carr <shane@unicode.org> Date: Tue Aug 15 18:57:06 2023 -0700 Forbid the empty string in AuxiliaryKey commit d0e30e2 Author: Shane F. Carr <shane@unicode.org> Date: Tue Aug 15 18:54:44 2023 -0700 Docs, tests, cleanup commit 06e453c Author: Shane F. Carr <shane@unicode.org> Date: Tue Aug 15 18:29:18 2023 -0700 Change custom error type to DataError::KeyLocaleSyntax commit 6cb45f6 Author: Shane F. Carr <shane@unicode.org> Date: Tue Aug 15 18:18:00 2023 -0700 Make comparison operations work commit 59aa0b4 Author: Shane F. Carr <shane@unicode.org> Date: Tue Aug 15 15:55:52 2023 -0700 Start writing impl FromStr for DataLocale commit 3d36855 Author: Shane F. Carr <shane@unicode.org> Date: Tue Aug 15 13:23:16 2023 -0700 Initial auxiliary key APIs

…sing U+D7FF

dpulls · 2023-08-18T22:47:55Z

🎉 All dependencies have been resolved !

skius · 2023-08-21T10:04:40Z

This fails CI because the sample transform rule directory (cldr-transforms-main) does not actually exist when downloading fresh CLDR data, how should I resolve that?

robertbastian · 2023-08-21T10:10:41Z

This fails CI because the sample transform rule directory (cldr-transforms-main) does not actually exist when downloading fresh CLDR data, how should I resolve that?

In https://github.com/unicode-org/icu4x/blob/main/tools/testdata-scripts/src/bin/download-repo-sources.rs#L121 you need to move cldr-transforms-main into a temp dir and recover it after.

robertbastian · 2023-08-22T13:01:32Z

experimental/transliteration/data/Cargo.toml

+    "README.md"
+]
+
+[dependencies]


nit: remove

robertbastian · 2023-08-22T13:03:51Z

experimental/transliteration/src/provider.rs

+#[derive(
+    Debug,
+    Clone,
+    zerofrom::ZeroFrom, // QUESTION: does ZeroFrom need to be behind a feature?


It's required by DataPayload::with_mut and DataPayload::try_map_project, which is used by .as_deserializing() only, so it could be behind the serde feature. However, it's a dependency of icu_provider, so making it optional doesn't gain much, and we don't have it behind a feature anywhere.

robertbastian · 2023-08-22T13:14:30Z

experimental/transliterator_parser/src/compile.rs

+#[doc(hidden)]
+pub fn legacy_id_to_internal_id(source: &str, target: &str, variant: Option<&str>) -> String {
+    // TODO(#3891): Decide representation for unknown BCP47 IDs
+    let mut id = format!("x-{source}-{target}");


praise: good solution for now

robertbastian · 2023-08-22T13:19:27Z

provider/datagen/src/transform/cldr/source.rs

+        let dir_suffix = self.0.dir_suffix()?;
+        let path = format!("{}-{dir_suffix}/main/{transform}/metadata.json", self.1);
+        if self.0.serde_cache.file_exists(&path)? {
+            self.0.serde_cache.read_and_parse_json(&path)
+        } else {
+            Err(DataErrorKind::Io(std::io::ErrorKind::NotFound)
+                .into_error()
+                .with_display_context(&path))
+        }


You should be able to construct a CldrDirLang on the fly to call its read_and_parse

I assume you meant CldrDirNoLang, because there is no language identifier around. Doing this now:

let cldr_dir = CldrDirNoLang(self.0, format!("{}-{dir_suffix}/main/{transform}", self.1)); cldr_dir.read_and_parse("metadata.json")

robertbastian · 2023-08-22T13:22:46Z

provider/datagen/src/transform/cldr/source.rs

+        let dir_suffix = self.0.dir_suffix()?;
+        let path = format!("{}-{dir_suffix}/main/{transform}/source.txt", self.1);
+        if self.0.serde_cache.file_exists(&path)? {
+            let buf = self.0.serde_cache.root.read_to_buf(&path)?;


Change read_to_buf by read_to_string, then the AbstractFs can handle UTF-8 errors. JSON and TOML deserialization can be done on strings as well.

Added read_to_string to AbstractFs

robertbastian · 2023-08-22T13:28:01Z

provider/datagen/src/transform/cldr/transforms/mod.rs

+    fn lookup_dir_from_internal_id(
+        &self,
+        internal_id: &str,
+    ) -> Result<Option<(String, bool)>, DataError> {


nicer to return icu_transliterator_parser::Direction

robertbastian · 2023-08-22T13:29:45Z

provider/datagen/src/transform/cldr/transforms/mod.rs

+
+fn find_bcp47_in_list(list: &[String]) -> Option<String> {
+    for item in list {
+        if item.contains("-t-") {


might be better to parse to Locale.

Parsing aliases as an enum over Locale (for -t-) and String (for legacy IDs) now

robertbastian · 2023-08-22T13:33:36Z

provider/datagen/tests/data/json/transliterator/rules@1/und+de-t-de-d0-ascii.json

+        ]
+      },
+      {
+        "cp_inv_list": [


oh these are quite big 😬

skius added 3 commits August 16, 2023 13:15

add sample transliterator data (de-ASCII)

8b12ce1

DatagenProvider for Transform Rules

b327e5e

add todos

7d74393

skius mentioned this pull request Aug 16, 2023

Checklist for Transliteration #3736

Closed

41 tasks

skius added 18 commits August 18, 2023 14:50

Merge branch 'main' into datagen-transform-rules

68bd038

hook up the parser to datagen

6edbd3c

add zerovec:databake dep

1c0de7c

clippy

9f267d4

use collections databake feature

ad73a31

add more sample transforms

0158d6b

add feature for cargo make testdata

521e52c

(wip) broken human-readable datagen due to InterIndic-Arabic's CPIL u…

432a2be

…sing U+D7FF

make data serialize to json

d4b3bb2

use available transliterator mapping in compilation

7590a32

regenerate testdata

69f8005

use visibility information

3de8cd5

regenerate testdata

181ced6

remove todo

8434e85

add todo

9cc37ec

add issue number to crate feature todo

83696c6

add Greek-Latin/BGN

1ee858f

skius added 7 commits August 19, 2023 02:06

Merge branch 'main' into datagen-transform-rules

84f9bef

fix tests and clippy

f78635b

add correct cfg for transform-specific features

ac6e142

add direct dependencies to datastruct

5804081

regenerate testdata

f596a56

add features to icu_transliteration

2d06388

add transliteration/compileddata?

98b605c

skius requested review from sffc, robertbastian and a team as code owners August 21, 2023 10:03

skius removed request for a team and Manishearth August 21, 2023 10:03

skius added 2 commits August 21, 2023 10:20

download-repo-sources workaround for transform rules

5d517fe

Debug for CPILULE

3ec878e

skius requested a review from echeran as a code owner August 21, 2023 14:31

skius removed the request for review from echeran August 21, 2023 14:32

Debug for CPILASLULE

299d17a

robertbastian previously approved these changes Aug 22, 2023

View reviewed changes

addressing first part of review

0cb15c3

skius dismissed robertbastian’s stale review via 0cb15c3 August 22, 2023 14:04

skius added 5 commits August 22, 2023 16:02

transform aliases parse to either Locale or String, normalize legacy IDs

273cb25

regenerate testdata

3b249bb

fix tests

0fa9929

Merge branch 'main' into datagen-transform-rules

6566d63

fix license

fb3a112

skius force-pushed the datagen-transform-rules branch from 57e1af3 to fb3a112 Compare August 22, 2023 17:28

robertbastian previously approved these changes Aug 22, 2023

View reviewed changes

regenerate baked data

1f9da83

skius dismissed robertbastian’s stale review via 1f9da83 August 22, 2023 17:44

robertbastian previously approved these changes Aug 22, 2023

View reviewed changes

skius added 2 commits August 22, 2023 22:40

Merge branch 'main' into datagen-transform-rules

7312e8c

regenerate testdata

bba758c

skius dismissed robertbastian’s stale review via bba758c August 22, 2023 20:41

robertbastian approved these changes Aug 23, 2023

View reviewed changes

robertbastian merged commit c5731f7 into unicode-org:main Aug 23, 2023
26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transliterator DatagenProvider #3877

Transliterator DatagenProvider #3877

skius commented Aug 16, 2023 •

edited

Loading

dpulls bot commented Aug 18, 2023

skius commented Aug 21, 2023

robertbastian commented Aug 21, 2023

robertbastian Aug 22, 2023

robertbastian Aug 22, 2023

robertbastian Aug 22, 2023

robertbastian Aug 22, 2023

skius Aug 22, 2023

robertbastian Aug 22, 2023

skius Aug 22, 2023

robertbastian Aug 22, 2023

robertbastian Aug 22, 2023

skius Aug 22, 2023

robertbastian Aug 22, 2023

Transliterator DatagenProvider #3877

Transliterator DatagenProvider #3877

Conversation

skius commented Aug 16, 2023 • edited Loading

dpulls bot commented Aug 18, 2023

skius commented Aug 21, 2023

robertbastian commented Aug 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skius commented Aug 16, 2023 •

edited

Loading