This May, we had the privilege of leading an engaging tutorial titled “Accelerating Generative AI with PyTorch” at the ISC High Performance 2024 conference. Hosted in the vibrant city of Hamburgh, our session offered a deep dive into the latest computational challenges and optimization techniques that are shaping the future of Generative AI.
The intermediate-level tutorial, designed for deep learning practitioners, was a collaborative effort, featuring contributions from NCC Cyprus, and partners from NCC Italy, NCC Bulgaria, NCC Netherlands, and the CASTIEL2 project. Each entity played a crucial role in bringing this initiative to life under the umbrella of the EuroHPC JU and the EuroCC2 project. Held at the ISC venue, the event spanned four hours of hands-on learning and dedicated talks. We explored PyTorch’s native features, demonstrating how they can be leveraged to overcome common yet complex issues in high-performance computing environments.
The tutorial was characterized by a high level of participant engagement. Attendees, equipped with basic to intermediate Python skills and some background in high-performance computing (HPC), had the opportunity to access and use a real supercomputer, kindly offered by the GWDG, in order to execute the hands-on exercises. In addition, the interactive format encouraged lively discussions and shared learning experiences, creating a rich dialogue among experts.
The topics covered in the tutorial included:
- Accessing an HPC System: Hands-on introduction on HPC systems and how users can access them.
- Profiling Guided Optimisation: Introduced the notion of profiling and how users can leverage it to optimise AI workloads.
- Distributed ML Models: Efficient scaling of machine larning models with PyTorch via Distributed Learning.
- Introduction to Large Language Models (LLMs) and Llama2.
- Optimising LLMs: Enhancing Efficiency and Scalability of LLMs with PyTorch, via techniques such as Just-in-Time (JIT) compilation, quantization of weights, model parallelism and speculative decoding.
- Overview of methods for efficient generative AI training and Inference.
The feedback received has been very positive, affirming the importance of such hands-on, practical tutorials in the ever-evolving field of artificial intelligence, and LLMs. The positive feedback from attendees has encouraged us to consider future topics and improvements for upcoming tutorials and already looking forward to repeating this next year.
EuroCC Cyprus would like to thank everyone who participated in and supported our tutorial at ISC High Performance 2024. It a great opportunity to share knowledge and a chance to grow our community of AI and HPC practitioners.
