GPU Programming

Learn high-level GPU architecture and how to build modern AI systems quickly and efficiently!

General topics:

• Intuition for how GPU hardware basics naturally prescribe the style of GPU software
• GPU performance modeling - AI programs are generally simple, and performance can often be understood with one or two formulas
• Triton programming language for efficiently implementing AI programs on a GPU
• How to write fast implementations for every part of a transformer
• How to adapt existing performant activation functions and models for your specific idiosyncratic use-case

Activities:

• Building spreadsheet models for the runtime performance of AI programs
• Guided exercises translating simple Python code using Pytorch for ML to run on GPUs using Triton

Prerequisites:

• Interest in thinking deeply about computer hardware is expected
• General coding experience is required
• Basic familiarity with the C programming language is helpful
• No experience with GPU programming or AI/ML techniques is needed
• Bring your own laptop

Sophia Wisdom taught herself GPU programming in 2022 after realizing AI was going to be a big deal. She worked at magic.dev on implementing their novel architecture on GPUs. Recently she has been selling data to AI companies and is working on a LLM inference engine.