Question bank

  1. A proxy-tuned learning rate makes the 8x wider model divergeScalingMultiple choiceHardFree
  2. One neuron fires on citations, dialogue, HTTP requests and Korean textInterpretabilityMultiple choiceMediumPro
  3. The reward keeps climbing, but people prefer an earlier checkpointRLHFMultiple choiceMediumPro
  4. Test accuracy sits under 10% for 24,000 steps, then climbs to 100%InterpretabilityMultiple choiceHardPro
  5. Your LLM judge flips its verdict when you swap the answersEvaluationMultiple choiceMediumFree
  6. After a learning-rate increase, loss spikes a few thousand steps inTraining stabilityMultiple choiceHardPro
  7. At a million tokens, one layer's activations don't fit on one GPUScalingMultiple choiceHardPro
  8. Ten hand-set weights add ten-digit numbers, but training needs 36ScalingMultiple choiceMediumPro
  9. 57% on grade-school math, no more training, lots of inference computeReasoningMultiple choiceMediumFree
  10. Your reward model picks right answers reached through wrong stepsReasoningMultiple choiceMediumPro
  11. More users, an idle GPU, and every user gets slowerInferenceMultiple choiceHardPro
  12. It knows who Daphne Barrington is, but can't say who directed her filmDataMultiple choiceEasyPro
  13. What forms inside a model at this bump in its loss curve?InterpretabilityMultiple choiceMediumPro
  14. The weights fit in INT8, but the activations don'tQuantizationMultiple choiceHardPro
  15. Pretrained on 4K tokens, now it has to read 32KPositional encodingMultiple choiceMediumPro
  16. DPO loss keeps dropping, but by epoch 3 the answers degradeFine-tuningMultiple choiceHardPro
  17. You switch training to fp16 and the gradients round to zeroTraining stabilityMultiple choiceMediumFree
  18. Why does attention divide every score by the square root of d?AttentionMultiple choiceEasyPro
  19. The rule says 160B tokens for an 8B model. Llama 3 used 15 trillionScalingMultiple choiceMediumPro
  20. RL on a 7B math model, and PPO wants a second 7B model as the criticRLVRMultiple choiceMediumPro
  21. 100B tokens of text in your language, and compute for 4 trillionDataMultiple choiceMediumPro
  22. As the model grows, test error goes down, then up, then down againScalingMultiple choiceHardPro
  23. 99% confident, but right only about 90% of the timeEvaluationMultiple choiceEasyPro
  24. Why does a 47B model run about as fast as a 13B one?ArchitectureMultiple choiceEasyPro
  25. FlashAttention recomputes in the backward pass and still runs fasterAttentionMultiple choiceMediumPro
  26. Why not let the draft model write twenty tokens at a time?InferenceMultiple choiceMediumPro
  27. Push the MoE load-balancing loss too high and the model gets worseArchitectureMultiple choiceEasyPro
  28. A chatbot runs fine for an hour, then outputs gibberishInferenceMultiple choiceMediumPro
  29. Aces 20-digit addition, fails at 100 digitsPositional encodingMultiple choiceEasyPro
  30. Checking that an LLM judge's position bias is goneEvaluationShort answerMediumPro
  31. Testing that a 32K-context model really uses the far endPositional encodingShort answerMediumFree
  32. Turning step labels into one score for a whole solutionReasoningShort answerMediumPro
  33. Fitting more users at the same tokens per secondInferenceShort answerHardPro
  34. Checking that a sparse autoencoder feature means one thingInterpretabilityShort answerMediumPro
  35. Picking the draft length for speculative decodingInferenceShort answerHardPro
  36. A 30-layer ReLU net with Glorot init never starts learningInitializationMultiple choiceEasyPro
  37. Cross-entropy turns NaN once a logit passes 89Training stabilityMultiple choiceEasyPro
  38. Names and compounds keep coming out as UNKTokenizationMultiple choiceEasyPro
  39. The 34-layer net has higher training error than the 18-layer oneArchitectureMultiple choiceEasyPro
  40. BatchNorm in a language model: eval loss depends on the batchNormalizationMultiple choiceMediumPro
  41. Dropout is off at inference and the activations come out twice as largeRegularizationMultiple choiceEasyPro
  42. A hand-written Adam blows up in the first hundred stepsOptimizationMultiple choiceEasyPro
  43. One batch in a few thousand blows the loss upTraining stabilityMultiple choiceEasyPro
  44. The top logit keeps pulling away from the restRegularizationMultiple choiceEasyPro
  45. Teacher-forced loss looks great, but one wrong word derails the captionOptimizationMultiple choiceMediumPro
  46. A word-level LSTM with 30M embedding parameters overfitsArchitectureMultiple choiceEasyPro
  47. Training accuracy hits 100% while validation never leaves chanceDataMultiple choiceMediumPro
  48. Choosing and checking the init for a deep ReLU netInitializationShort answerMediumPro
  49. Writing a cross-entropy kernel that survives fp16 logitsTraining stabilityShort answerHardPro
  50. Setting the gradient-clipping threshold for a new runTraining stabilityShort answerMediumPro
  51. Explaining why a language model shouldn't use BatchNormNormalizationShort answerMediumPro
  52. An L2 penalty under Adam barely shrinks the high-gradient weightsOptimizationMultiple choiceMediumPro
  53. Beam search loops, full sampling derails into the tailInferenceMultiple choiceMediumPro
  54. A Post-LN Transformer trains far worse without warmupTraining stabilityMultiple choiceMediumPro
  55. The right document is retrieved, but the model misses it in the middleLong contextMultiple choiceMediumPro
  56. The token budget grew mid-run, but the cosine already planned its decayScalingMultiple choiceMediumPro
  57. Batch 256 to 8,192 on 256 GPUs, and accuracy dropsOptimizationMultiple choiceMediumPro
  58. 120 GB of identical optimizer state on every one of 64 GPUsDistributed trainingMultiple choiceMediumPro
  59. Raise the distillation temperature and the soft loss stops matteringDistillationMultiple choiceMediumPro
  60. The SVM beats your logistic regression on the metric you shipOptimizationMultiple choiceMediumPro
  61. The loss is 41 before the first update on a 32K vocabularyInitializationMultiple choiceEasyPro
  62. Every step rewrites 5% of each layer's weightsOptimizationMultiple choiceMediumPro
  63. A linear probe whose weights never stop growingRegularizationShort answerMediumPro
  64. At 32K context, tokens per second fall to a thirdScalingMultiple choiceMediumPro
  65. How long will 300B tokens take on 64 H100s?ScalingMultiple choiceEasyPro
  66. Picking the model size for a fixed compute budgetScalingShort answerHardPro
  67. 40B tokens of instruction data, and SFT on all of it costs knowledgeDataMultiple choiceMediumPro
  68. Teaching a left-to-right code model to fill in the middleDataShort answerHardPro
  69. The summarization SFT loss falls fast, and the summaries barely changeFine-tuningMultiple choiceEasyPro
  70. After SFT on expert answers, the model invents more specificsFine-tuningMultiple choiceMediumPro
  71. A single LoRA fine-tune decodes slower than its base modelFine-tuningMultiple choiceEasyPro
  72. Choosing the KL strength for a DPO runRLHFShort answerMediumPro
  73. A PPO agent with a good critic still learns erraticallyPolicy gradientsMultiple choiceMediumPro
  74. GRPO on math, and the wrong answers keep getting longerRLVRMultiple choiceHardPro
  75. Filtering RL prompts with a reward-model thresholdRLHFMultiple choiceMediumPro
  76. Checking the reward function before math RLRLVRShort answerMediumPro
  77. Best-of-16 with a 6,000-token prompt, and prefill eats the budgetInferenceMultiple choiceEasyPro
  78. INT3 weights more than double the perplexityQuantizationMultiple choiceHardPro
  79. Half the weights pruned, and no speedupInferenceShort answerMediumPro
  80. RL wins at pass@1 but the base model wins at pass@256RLVRMultiple choiceHardPro
  81. A small high-quality math set and the learning rate is about to decayDataMultiple choiceMediumPro
  82. Twice the SFT data from the same source, and the answers are no betterFine-tuningMultiple choiceMediumPro
  83. A long prompt arrives and every running stream stallsServingMultiple choiceHardPro
  84. GRPO's samples go identical and accuracy stalls early in trainingRLVRMultiple choiceHardPro
  85. Loss near zero in an hour, but generation is gibberishAttentionMultiple choiceEasyPro
  86. Labelers prefer the RLHF model, but SQuAD and DROP scores fellRLHFMultiple choiceMediumPro
  87. Extending to 128K on mostly books, and short-text loss risesLong contextMultiple choiceMediumPro