Training Cost Console

The cost of training a large model collapses, with surprising accuracy, into a single relationship: about six floating-point operations per parameter per token. Two for the forward pass, four for the backward — multiply by the parameter count and the number of training tokens and you have the total compute, the 6ND rule that underpins every training budget. It turns the two decisions that define a model, how big and how much data, directly into FLOPs and therefore dollars.

But FLOPs aren't time, because no cluster runs at its peak. Model FLOPs Utilization — the fraction of theoretical peak actually achieved after communication overhead, memory limits, and pipeline bubbles — is the efficiency tax, and it's large: real runs land at 30–50%. It scales the schedule and the cost linearly, so a run at 40% MFU takes two and a half times longer than the peak number suggests. Using a realistic MFU rather than the spec-sheet FLOPS is the single most important thing in an honest estimate, which is why it's a first-class input here.

At frontier scale these numbers become staggering. Hundreds of billions of parameters on tens of trillions of tokens is upward of 10²⁵ FLOPs, demanding tens of thousands of GPUs for months and tens of millions of dollars in compute alone. And that compute is only the floor — the full budget adds data pipelines, engineering teams, checkpoint storage, and the failed runs that can add a half again to the total. The compute figure is just the part that scales most violently with ambition.

It's also an energy and carbon story now. Tens of thousands of kilowatt-class GPUs for months is gigawatt-hours and hundreds of tonnes of CO₂, varying several-fold with how clean the grid is — increasingly a reported number. Once you have the compute cost, decide whether to rent or buy it in the Accelerator ROI console, and size the cluster in the GPU Cluster Sizing console.