Your manager asks how long it will take to train a 3B-parameter model on 300B tokens with 64 H100s.
Each H100 peaks at about 989 TFLOP/s in dense bf16, and your stack reaches 40% MFU at this scale.
Your manager asks how long it will take to train a 3B-parameter model on 300B tokens with 64 H100s.
Each H100 peaks at about 989 TFLOP/s in dense bf16, and your stack reaches 40% MFU at this scale.