Interview prep for AI research engineers
87 questions from AI lab interviews, with worked answers and short video explanations.
Mistral AI
You switch training to fp16 and the gradients round to zero
Mistral AI
You move a training run from fp32 to fp16 to halve activation memory and use tensor cores; the forward pass looks fine, but the loss eventually diverges. Many backward-pass activation gradients are below about , the smallest positive fp16 value, so they round to zero.
What would you change so those gradients survive in fp16?
Formats
Multiple choice
Four options, one best, and why each of the other three fails.
Short answer
The interviewer's follow-up, checked against a model answer and its key points.
Video
The setup and each option, shown in 3D.
Where the questions come from
Our academic partnerships, people in our network who've interviewed at AI labs, university courses and their exams, industry partners, and interview reports on Glassdoor and Reddit. We rewrite each question so it stands on its own, with a worked answer.
Free questions
All 87 questionsResources
Courses, textbooks and guides, from Karpathy's Zero to Hero to Stanford CS336 and DeepMind's scaling book.
See all 31