Adding Megatron models.

[Narsil]

🌟 New model addition

Is it feasible to add Megatron models ? It seems the architecture is really just a GPT2, most of the work should be in creating the config, fusing layers from the available weights here: https://github.com/pytorch/fairseq/tree/master/examples/megatron_11b and making them available.

There are Nvidia's megatron (Bert and Gpt variants) and Facebook-11b megatron (gpt variant)

If we stick to that then we can't run the model on a single GPU, so we should probably make sure this is compatible with:

• [docs] outline sharded ddp doc #9208
• [trainer] deepspeed integration #9211

Is keeping the current GPT2 architecture and using deepspeed's ZeRo and other parallelism schemes without touching original implementation feasible?

Model description

https://github.com/pytorch/fairseq/blob/e3c4282551e819853952284681e9ed60398c5c4a/examples/megatron_11b/README.md

Open source status

• the model implementation is available: https://github.com/ngoyal2707/Megatron-LM/blob/adb23324c222aad0aad89308e70302d996a5eaeb/mpu/transformer.py (Most of the work seems to be on Matrix parallelization)
• the model weights are available: https://dl.fbaipublicfiles.com/fairseq/models/model_parallel/megatron_11b.tar.gz (Megatron 11b), https://github.com/NVIDIA/Megatron-LM#downloading-checkpoints (Nvidia's version, 3b and 8.3b don't seem to be available)
• who are the authors: (mention them, if possible by @gh-username) Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro https://arxiv.org/abs/1909.08053

https://developer.nvidia.com/blog/language-modeling-using-megatron-a100-gpu/

@stas00 @patrickvonplaten

[stas00]

Since DeepSpeed both integrates and uses Megatron-LM almost everywhere in its tutorials it most likely should just work. Of course, the devil is in the detail.

As I haven't had a chance to study/work with GPT2 yet, I will let others comment on the more important part of your query.

[jordiae]

Any plans of adding MegatronT5? (https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/model/t5_model.py)

[stas00]

As this is a really old thread, perhaps make a request in a new Issue, @jordiae?

And of course, if you're interested you're more than welcome to try and add it yourself. This is of course only an invitation.

[jordiae]

As this is a really old thread, perhaps make a request in a new Issue, @jordiae?

And of course, if you're interested you're more than welcome to try and add it yourself. This is of course only an invitation.

Got it! Posted here because the issue was open. Thanks.

[Narsil]

Will close this issue as it's really kind of outdated.