On Mistral-7B, the fraction of weights that can move at all

Quiet-STaR 1×10⁻⁶
More controls
jump to

A weight can move only if the step (≈ lr) is at least half the gap to its bf16 neighbour: lr ≥ 2^(⌊log₂|w|⌋ − 8).