Ray AI Libraries on TPU: Practical Setup Guide

Key points

• 1

  Ray Serve on TPU: Ray Serve handles multi-TPU models by requiring a `topology` field in `accelerator_config` (e.g., "4x4" for 16 chips). Without this, tensor-parallel models scatter across slices, causing jobs to hang indefinitely. For example, a Llama 3 8B model on 16 TPU chips needs `topology: "4x4"` to stay on one slice. Deploy with RayService on a vLLM TPU image, wait for `Running` status, then curl the endpoint. This avoids the common bug where missing topology causes workers to fail silently for hours.

• 2

  Ray Data on TPU: Ray Data's `iter_jax_batches()` API feeds JAX-native, device-sharded batches directly to TPU training without host-side NumPy-to-JAX copies. It handles ragged batches via `drop`, `pad`, or `raise` options. For instance, training a model with `ray.data.read_parquet()` uses this to process large datasets efficiently. This solves the bottleneck where naive data loaders choke TPU performance, especially for batch inference or training with JAX.

• 3

  Ray Train on TPU: JaxTrainer simplifies distributed training on TPU by taking a `topology` field (e.g., "4x4") instead of manual slice coordination. The training loop runs inside a worker function with `jax` imported at module scope (not top-level), ensuring each TPU initializes correctly. For example, a DPO training job uses `ScalingConfig(topology="4x4")` to wire workers into a single mesh. This eliminates complex hand-written code for cross-slice communication, enabling fault-tolerant training on preemptible TPU resources.