Markup-Tag-Evaluation implements metrics to evaluate hypothesized markup tag placements in a sentence against reference tag placement in the same sentence.
The metrics assume that each tag of the reference appears exactly once in the hypothesis. Chapter 3 of the paper Automatic Bilingual Markup Transfer provides a detailed description of the evaluation metrics.
pip install .
evaluate-markup-tags --help