Home > TERATEC FORUM > Workshops > Workshop 8

Teratec 2022 Forum
Wednesday, June 15 - Technical Workshop

Workshop 08 - 2:00 to 5:30 pm

High performance AI in the industry
Chaired by Cristel Saudemont, France Director, Supercomputing & AI , Higher Education and Research and Frédéric Parienté, senior manager in the solutions architecture and engineering group, Nvidia

Practical lessons from training extreme-scale language models for the multilingual Muse API
By Julien Launay, PhD Student at ENS/Machine Learning Research Scientist, LightOn

Large language models have gained increased traction in the research community, but they remain only scarcely deployed at scale in the industry. In this talk, we go over how we are bringing large models with our Muse API to French, Arabic, Spanish, German, and English. We first discuss the importance of data collection and curation, and how it can be tackled at scale. We then highlight how an efficient training codebase based on Megatron+DeepSpeed is key to efficiently bringing these models to life. Finally, we review some use cases for these models, and how we can allow for efficient task specialisation at scale.

Biography: Julien Launay is the lead of the extreme-scale project at LightOn, working to extend the capabilities and the availability in various languages of large language models.

Register now and get your badge here

© Ter@tec - All rights reserved - Lawful mention