ScalingFreeYou sweep the learning rate on a small proxy and widen the model 8x for the real run. At the proxy's best rate the wide model diverges, and a full-width sweep is too expensive.
The model uses standard fan-in initialization and one learning rate for every layer at every width.