How elastic training recovers from TPU failures without restarting

Key points

• 1

  Why distributed training fails when a TPU dies: In traditional distributed training, each machine holds a piece of the model. When one TPU fails, the all-reduce operation (which syncs model updates) times out because the missing machine can't send data. This causes the entire training job to crash, requiring a full restart from the last checkpoint. The result is significant downtime: scheduling new pods, restarting processes, and reinitializing data loaders. For example, a failed TPU in a 48-chip cluster could take minutes to recover, as seen in the demo where the job took ~2 minutes to restart after a failure. This is especially costly for large models where even a small delay can add hours of training time.

• 2

  How elastic training works with MaxText: Elastic training uses three key components: Pathways (for orchestration), Orbax (for checkpointing), and the elastic_retry decorator (for failure handling). When a TPU fails, Pathways detects the issue as a Python exception (like a DATA_LOSS error), the elastic_retry decorator catches it, and Orbax ensures the last complete checkpoint is restored. This allows the training to continue without restarting the entire job. For instance, in the demo, the system recovered from a failed TPU in under 2 minutes by swapping only the affected slice while keeping the rest of the training process running. The single controller process (a Python script on a CPU node) sees all TPUs as local, enabling this seamless recovery without reinitializing the entire job.

• 3

  Key settings for elastic training: To enable elastic training, set these flags in MaxText: enable_single_controller=True (to run one Python process), elastic_enabled=true (to activate the retry), elastic_timeout_seconds=300 (how long to wait for a replacement TPU), and elastic_max_retries=10 (how many failures to tolerate). The elastic_min_slice_count flag controls whether to pause and resume (default: all slices) or scale down (1-47 slices). For example, in the demo, the system used checkpoint_period=100 steps to ensure frequent enough checkpoints for recovery. If a slice fails during a checkpoint write, MaxText exits to prevent partial checkpoints, which is why the checkpoint interval must balance between step time and failure recovery.

What changed