Optimizing Qwen 3.5 MoE on Ironwood TPU with Modular Engineering

Key points

• 1

  Qwen 3.5's Sparse Architecture: Qwen 3.5 uses a 400B parameter model with only 17B active parameters per token (4.3% activation ratio), enabling efficient inference. This sparse routing scheme means the model delivers 400B-class capacity while maintaining 20B-class speed. For example, a 248,320-token vocabulary with 60 layers and 4096 hidden dimensions requires careful handling of its hybrid layers—75% Gated DeltaNet (GDN) layers for linear attention and 25% Grouped Query Attention (GQA) layers. This architecture demands specialized sharding to avoid memory bottlenecks, as standard tensor parallelism would create fractional head sharding (e.g., 2/8 = 0.25 heads per device) that hardware can't handle. To optimize, engineers must target Qwen's unique components like GDN and GQA, which affect how data flows through the system.

• 2

  Hybrid Sharding for Hardware Efficiency: Traditional sharding methods fail with Qwen 3.5's 2 KV heads in GQA and 512 experts in MoE. Attempting 8-way tensor parallelism (TP=8) for GQA forces 0.25 heads per device, which is physically impossible. Instead, the team implemented 8-way data parallelism (DP=8) for attention and 8-way expert parallelism (EP=8) for MoE layers. This avoids duplicating KV cache memory and ensures each device processes full 2 KV heads locally. For example, in prefill-heavy workloads (8K input / 1K output), this sharding strategy prevents the system from being capped at ~200 concurrent requests—instead achieving 512 concurrency. The solution uses a custom hierarchical reduce-scatter to handle token outputs across devices, reducing communication latency by 6x through intra-chip shared memory transfers and inter-chip hypercube algorithms.

• 3

  Performance Gains from Targeted Optimizations: The modular approach delivered 3.1x speedups for decode-heavy workloads (1K input / 8K output) and 4.7x for prefill-heavy workloads (8K input / 1K output) at 512 concurrency. These gains come from specific optimizations: the 3-to-2 All-Gather optimization reduced routing metadata collective latency by half by stacking expert indices and topk weights into a single 32-bit integer array, and the hierarchical reduce-scatter improved token output handling by 6x through pipelined micro-batches. For instance, in a 64-concurrency run, the system processed 400GB of model weights without OOM errors by slicing data into 2-4 micro-batches, allowing TensorCore to hide communication latency behind compute. This means enterprises can migrate to Qwen 3.5 without legacy software barriers, using open-source frameworks like vLLM and SGLang for seamless deployment.

What changed