🤗 Hugging Cast S2E1 - LLMs on AWS Trainium and Inferentia!

The script opens with a warm welcome to the second season of Hugging Cast, an interactive live show focused on building AI with open models and open source. The host expresses excitement about returning and acknowledges the audience, including those who have participated in previous shows. The new season aims to balance new news with practical demos, allowing viewers to learn and apply AI use cases in their companies. The host emphasizes the goal of maintaining the show's live and interactive nature, with plans to take audience questions after 30 minutes of demos. The show's intention is to collaborate with various partners to build an open platform for easy use of models and libraries on any compute stack, highlighting partnerships with cloud platforms like AWS, Google Cloud, and Azure, as well as hardware platforms like Nvidia, Intel, AMD, and on-prem platforms like Dell and IBM.

This paragraph delves into the specifics of using Hugging Face on AWS, particularly focusing on AWS custom silicon, which includes instances like Tranium and Inferentia, designed for AI workloads. The host explains the collaboration between Hugging Face and AWS engineers to streamline model usage on these instances. The Optimum Neuron library is introduced as a bridge between Hugging Face models and the hardware stack of Tranium and Inferentia, designed to simplify the process of leveraging these accelerators. The benefits of using Inferentia 2, including speed and cost savings, are discussed, with examples of how it can be used for model training and inference. The paragraph also covers the availability of comprehensive documentation and resources for using Optimum Neuron and AWS instances, as well as the potential for model transfer between different hardware types.

The host introduces the first demo by Phillip, which showcases deploying large language models on Inferentia 2 using Text Generation Inference (TGI). The demo is contextualized by discussing the capabilities of the Optimum Neuron documentation and the availability of tutorials for various models, including sentence Transformers on AWS Inferentia. The host also touches on the ease of deploying models to endpoints for integration with other applications. The paragraph outlines the different instance sizes available for Inferentia 2 and their respective pricing, highlighting the cost-effectiveness of using these instances for AI tasks. The benefits of TGI, including its streaming capabilities, are emphasized, providing a more dynamic interaction experience compared to traditional HTTP endpoints.

This section provides a detailed walkthrough of deploying a model using TGI on AWS Inferentia. The host discusses the installation of necessary packages like Sagemaker and Transformers, the acquisition of necessary permissions for AWS operations, and the retrieval of the TGI container image. The process of compiling the model for Inferentia using the Optimum CLI is explained, with attention to parameters such as batch size, sequence length, and number of courses. The host also mentions the collaboration with AWS to create a public cache for popular models, reducing the compilation time for users. The deployment of the Sapphire 7B model on Inferentia is used as an example, and the host demonstrates how to run inference and handle streaming responses for immediate feedback.

Mikel takes over to discuss the training of large language models (LLMs) on Tranium instances using Optimum Neuron. He emphasizes the importance of understanding memory requirements for model training, including model weights, gradients, optimizer state, and activations. The challenges of fitting large models into device memory are addressed, and various parallelism methods are introduced as solutions, including data parallelism, tensor parallelism, and pipeline parallelism. Mikel explains how these methods can be integrated into Optimum Neuron to allow for the training of larger models. He provides a simple code snippet to demonstrate the ease of use of Optimum Neuron for training, highlighting that users do not need to understand the technical details of parallelism methods to take advantage of them. The availability of resources and documentation for setting up AWS Tranium instances and running training scripts is also mentioned.

The session concludes with a recap of the key points discussed in the episode, including the deployment and training of large language models on AWS Inferentia and Tranium instances using Hugging Face's Optimum Neuron. The host expresses gratitude to the guests and the audience for their participation and sets expectations for the next episode, which will focus on using Hugging Face models in different computing environments. The host also invites audience questions and addresses a few, such as the use of pipeline parallelism for memory fitting and speed-up, the support for multimodal LLMs, and the potential for parameter-efficient fine-tuning methods like PFT on Tranium. The episode ends with a reminder of the show's return in about a month and an encouragement for audience interaction.