Scripts for preprocessing and experiments for the paper Automatic Bilingual Markup Transfer.

Dependencies

• Python3
• localization-xml-mt dataset
• SentencePiece
• FastAlign to generate alignments

Usage

# Run preprocessing, outputs written to ./bpe and ./text
export LOCALIZATION_XML_DIR="/your/path/localization-xml-mt"
./preprocess/run.sh

# Run FastAlign, outputs written to ./align/fastalign
export FASTALIGN_DIR="/your/path/fast_align"
./align/run_fastalign.sh