NVIDIA Proved 4-Bit Training Works at Real Scale (Not Just Inference)

A 12-billion-parameter Mamba-Transformer, trained on 10 trillion tokens, entirely in 4-bit floating point, matched an FP8 baseline almost…