Feature request: Summary data of each language including rows with metadata, gender, age, accent distribution

[KathyReid]

Firstly a huge thanks to the team for all the efforts that go into Common Voice, hugely appreciated.

If possible, I would like summary data for each language for which a dataset is released, showing:

• Unique rows
• Unique contributors
• Rows with metadata (#)
• Rows with metadata (%)
• Approximate hours with metadata
• Row count by genders
• Row count by age ranges
• Row count by accents

This allows a researcher to understand easily how much of a language dataset has metadata, and what the metadata distribution looks like. Some of this is already in the JSON files in this directory, but this is a different "slice and dice" of the summary data.

I have a Python script which calculates these based on the validated.tsv file of the language's dataset, happy to share.

Kind regards,
Kathy

[mozgzh]

I think this is a great idea - if you're willing, please do share the code as I'm sure others would be interested. This repo (cv-dataset) is housing the compiled statistics, CorporaCreator is where the actual logic lives that calculates splits. Since the nature of the data is fluid and users would likely want to calculate their own statistics, it might make sense to have a separate directory to house scripts that run outside of the CorporaCreator flow.

[HarikalarKutusu]

I needed similar data in all of the following stages:

• Validated
• Official splits
• Custom splits (CorporaCreator -s option)
• After elimination/import (e.g. Coqui after import_cv.py as some of the data got filtered out)

The last one is really what is needed.

Therefore having a set of separate codes that can be run on demand would be useful.

[HarikalarKutusu]

For those reading this, @KathyReid has been doing it 🥇

https://observablehq.com/@kathyreid/mozilla-common-voice-v11-dataset-metadata-coverage

Bülent