2023-05-03 AI Training Series - Introduction to the LRZ AI Systems (hdta3s23)

Contents

The aim of this course is to give an overview of the LRZ AI Systems, and provide participants with the knowledge and skills necessary to efficiently utilise them. The course consists of mini lectures, demos and hands on sessions (breaks included) covering the following topics:

• Resources overview of the LRZ AI Systems
  

• Fundamentals of Deep Learning

• Distributed Training of Neural Networks

Three blocks of content, devoting roughly an hour each to the first two, and two and a half hours to the third one:

• Overview of LRZ AI Systems (1h)
  • Hardware overview (B)
  • Access mode for the different resources (B)
  • Execution Mode (software stack) (B) + (I)

• Fundamentals of Deep Learning (1h)
  • Introduction to Neural Networks (B)
  • Training Neural Networks (B)
  • Introduction to Convolutional Neural Networks (B)
  • Introduction to Transformers (B)
  • Exercises: Training Convolutional Neural Networks and Transformers on GPUs (I)

• Distributed Deep Learning Training Part I (1h)
  • Motivation for Distributed Deep Learning Training (B)
  • Overview of Techniques for Distributed Deep Learning Training (B)

• Distributed Deep Learning Training Part II (1.5h)
  • Data Parallelism (I)
  • Exercise: Data Parallelism with Horovod (I)
  • Exercise: Data Parallelism with Pytorch Distributed Data Parallel (I)  
  • Model Parallelism - Pipeline Parallelism and Tensor Parallelism (A)
  • Exercise: Pipeline Parallelism with Pytorch Pipe (A)

Prerequisites

• AI Training Series: Orientation Session (or comparable previous knowledge)
• AI Training Series: Introduction to Container Technology & Application to AI at LRZ (or comparable previous knowledge)

Hands-On

During the course a live demo on how to access and operate the LRZ AI system will be showcased. In addition, the parallelisation of the training of a ML model will be also demonstrated. 

Content Level

The content level of the course is broken down as:

Language

English

Lecturers

Maja Piscac (LRZ)

Prices and Eligibility

The course is open and free of charge for academic participants from Germany.

Registration

Please register with your official e-mail address to prove your affiliation.

Withdrawal Policy

See Withdrawal

Legal Notices

For registration for LRZ courses and workshops we use the service edoobox from Etzensperger Informatik AG (www.edoobox.com). Etzensperger Informatik AG acts as processor and we have concluded a Data Processing Agreement with them.

See Legal Notices