Insight about the dataset size

[Kin-Zhang]

Please check this issue for more: opendilab/InterFuser#3, wayveai/mile#4

Questions: How to prove/analyze whether it's the method or the large data that brings the boost to the performance? if the dataset size is 6x-10x larger than others.

---

Table about the dataset size (copy the whole dataset size frames) and DS score, I linked each method with their GitHub link here:

Methods	Dataset size (Total frames)	DS Score*	Link
LAV	399K [399776]	61.85	CVPR'22 LAV
TCP	420K	75.14	NIPS'22 TCP
CVPR Transfuser	150K	16.93	CVPR'21 Transfuser
PAMI Transfuser	228K	61.18	PAMI'22 Transfuser
InterFuser	3M = 3, 000K	76.18	CoRL'22 InterFuser
MILE	2.9 M =2, 900K	x	NIPS'22 MILE

*: only count in the online leaderboard, not paper result table here, x means no data in the online leaderboard

Hint: 1M = 1,000 K

If you noticed other methods, please leave a comment, I will update it here.

---

This message is left for people who want to do this task: architecture and methods are important, but for this task, the way to collect dataset (expert), dataset size should be also considered when making the comparison.

[Kin-Zhang]

Screenshot from the paper <PlanT: Explainable Planning Transformers via Object-Level Representations> as Kait0 mentioned:

This may lead us to think more: the dataset size that 10x likes 2-3M with others' methods may also achieve such high scores or even higher (maybe).

Recent NIPS work <Model-Based Imitation Learning for Urban Driving> also uses the 2.9M dataset size, I linked their issue also:
wayveai/mile#4