Challenge
A tight training window and checkpoints that paused the cluster for minutes.
Solution
1,536 H200s racked in eleven days, with asynchronous checkpoints to hall-local flash.
Result
Checkpoints every 20 minutes, no stalls, the release shipped on schedule.
A deadline and a storage problem
Photon builds multimodal models that read documents, charts and images together. In spring 2026 the team had a release date, a model design and no cluster large enough to train it in time.
They also had a lesson from their last run: checkpoints that paused the whole job for minutes, so the team saved progress only twice a day and lost hours whenever a node failed.
“We had the model. What we needed was a cluster that was ready when we were.”
Eleven days to first job
Nordlys had H200 capacity on the floor in Hall A. The pod was cabled, burned in and handed over eleven days after signing, with Photon's own images and scheduler config already loaded.
What made checkpoints invisible
Shards copied to host memory in seconds, then written in the background.
A flash tier inside the hall, on a storage network separate from the compute fabric.
Large packed files per node instead of hundreds of thousands of small ones.
The release
Photon trained with a checkpoint every 20 minutes and recorded no checkpoint stalls across the run. Two node failures cost less than half an hour of work in total, and the model shipped on the planned date.
