From LR to ELR: A Better Heuristic for Pretraining Dynamics
Understanding pretraining dynamics is crucial for designing effective training hyperparameters for large language models (LLMs), particularly the learning rate (LR). However, LR does not always reflect how much the function represented by the model actually changes, and the optimal LR often shifts with model and data scale. In this post, we identify that the effective learning rate (ELR), which controls the directional changes of model weights, is a more intrinsic heuristic than LR as a tunable hyperparameter. Specifically, ELR delivers more accurate loss prediction under the multi-power law (MPL) model and transfers more reliably across model scales. More broadly, the ELR perspective guides the design of better schedules, which outperform conventional LR-schedule baselines.