Run Ray on TPU with Official Support

Key points

• 1

  TPU Slices Require Special Handling: TPUs are wired into fixed groups called slices (e.g., 4x4 topology = 16 chips across 4 hosts). Unlike GPUs, slices need all workers to stay within one slice to avoid communication failures. Ray handles this automatically via GKE's TPU webhooks that label slices, and Ray Core's slice_placement_group ensures workers stay on the same slice. For example, if you train a model with Ray on TPU, you must specify a topology like '4x4' in your manifest to ensure all workers stay within one slice—splitting workers across slices causes training to hang. This means you don't write custom placement code; Ray's API handles the slice reservation automatically for you.

• 2

  GKE and Ray Operator Setup: To run Ray on TPU, you use GKE with the Ray Operator add-on. The command `gcloud container clusters create CLUSTER --addons=RayOperator` installs two key components: KubeRay (the Kubernetes operator for Ray) and the TPU webhook (which labels hosts with slice information). You then define a node pool with `--tpu-topology=4x4` and `--num-nodes=4` to request a 4x4 slice. This setup ensures GKE provisions the slice, the webhook labels it, and Ray Core reserves the slice atomically. For instance, a single `ray.get(spg.placement_group.ready())` call reserves a 4x4 slice, and your workers automatically run on the same slice without manual intervention—no extra code is needed beyond specifying the topology in your manifest.

• 3

  Ray Core Handles Slicing Automatically: Ray Core's `slice_placement_group()` API (marked alpha) reserves entire slices atomically. In practice, you rarely call this directly—Ray AI libraries like JaxTrainer and Ray Serve handle slice placement for you. For example, when training with JaxTrainer, you declare a topology (e.g., '4x4') and Ray automatically schedules workers within the same slice. This means your existing code works unchanged on TPU; you only need to adjust your cluster manifest to specify the topology. If you're using Ray Data or Serve, you don't need to write slice placement code—Ray handles it behind the scenes, ensuring your model trains without communication failures.

What changed