Checkpointing 1,024 GPUs in under 40 seconds

On a 1,024-GPU pod, every second the cluster spends writing a checkpoint is a second nobody trains. In January a full 70B checkpoint paused our customers' jobs for close to four minutes. Today the same checkpoint blocks training for 38 seconds.

Where the four minutes went

We traced a dozen checkpoints end to end. Very little of the time was spent moving bytes. Most of it went to every rank waiting on the slowest writer, and to metadata calls queuing on a shared parallel file system that was also serving datasets.

• Stragglers. One rank on a busy node set the pace for all 1,024.

• Metadata. Hundreds of thousands of small shard files hit a single metadata service at once.

• Contention. Data loaders and checkpoint writes shared the same storage paths.

Asynchronous by default

The fix was to take writes off the critical path. Each node now copies its shard from GPU memory into host memory, which takes seconds, and training resumes. A background process then streams the shard to a flash tier inside Hall A over a storage network that is separate from the compute fabric.

Shards are packed into a few large files per node, so the metadata load fell by two orders of magnitude. Replication to a second storage room happens after the write is acknowledged, not before.

What it costs

Each node keeps one extra checkpoint in host memory, and the flash tier is sized at about three full checkpoints per pod. In return, teams can checkpoint every 20 minutes instead of every two hours, and a failed node costs minutes of work rather than an afternoon. The tooling ships with every Dedicated pod at no extra charge.