How to Train Really Large Models on Many GPUs?
<!-- How to train large and deep neural networks is challenging, as it demands a large amount of GPU memory and a long horizon of training time. This post reviews several popular training parallelism paradigms, as well as a variety of model architecture and memory saving designs to make it possible to train very large neural networks across a large number of GPUs. --> <p><span class="update">[Updated on 2022-03-13: add <a href="#ec">expert choice routing</a>.]</span><br/> <span class="update">[Updated on 2022-06-10]: <a href="https://gregbrockman.com/">Greg</a> and I wrote a shorted and upgraded version of this post, published on OpenAI Blog: <a href="https://openai.com/blog/techniques-for-training-large-neural-networks/">“Techniques for Training Large Neural Networks”</a></p>