Google Open-Sources TPU Sync, a C++ KV-Cache Transfer Library for TPUs; Its Own Low-Concurrency Tests Report Average TTFT Cuts of 60% and 90%

Overview

Google Cloud has released TPU Sync, which its blog post, dated November 9, 2026, calls an “open-source, native C++ library”. The post, by Fenghui Zhang (Distinguished Engineer, Google Cloud) and Weiren Yu (Senior Product Manager, Google Cloud), says the library moves data between TPU memory, host memory, shared storage and the data center network, and that it is aimed at inference and reinforcement learning (RL). The figures Google gives for it are its own, from internal tests run in July 2026. The New Stack reported the release on the same day, under a headline that adds “but there’s a catch”.

What the library does

According to the Google post, TPU Sync turns “host DRAM, local SSDs, and Google Cloud Managed Lustre” into a multi-tier key-value (KV) cache for inference. For RL, it exposes DMA and data-center-network primitives that higher-level frameworks can use to stream updated weights from trainers to samplers. The post names integrations with PyTorch through TorchTPU and with JAX, and says TPU Sync serves “as the TPU KV cache connector for vLLM, SGLang, and llm-d disaggregated serving”.

The project’s GitHub repository says “TPU Raiden codebase has graduated into TPU Sync”, which indicates the codebase was earlier called TPU Raiden. Its LICENSE file is the Apache License, Version 2.0, as read through the GitHub API on November 10.

Google labels some parts experimental. Among them, the post says the RL pipeline support is “included in the repository as an experimental module”, and it describes the tiering to shared secondary storage as experimental. The repository README adds that the PyTorch wheel’s “release process is not yet gated on end-to-end performance tests”.

Google’s reported figures

All numbers in this section are Google’s. The post’s footnote says they come from “internal testing conducted in July 2026 on TPU v7x (2x2x1 topology)” using “tpu-inference v0.24.0 (using JAX 0.10.2 and libtpu 0.0.42.1)”, and that results “will vary based on system configuration, network bandwidth, concurrent traffic, and specific user workload patterns”. By the Herald’s own comparison, JAX 0.10.2 is a version the README lists as supported, while the README’s default build version is 0.11.2.

In the results list, Google reports:

• Under the heading “Over 90% KV transfer overhead cut”, the post says that for Qwen3 235B, median KV transfer latency “dropped from several seconds down to a flat ~230 ms”.
• An average TTFT reduction of 60% for Qwen3 235B “under controlled, low-concurrency testing benchmarks”, and an average of 90% for the 480B model under low concurrency.
• An average increase of 30% to 70% in token output throughput for the 235B model during sustained traffic spikes, depending on traffic distribution.
• With prefix caching, average TTFT reduced “by up to 90% on 64K-context workloads”, with over 50% higher throughput and a 30% reduction in P99 inter-token latency.

The post’s introduction states the claim more broadly, saying TPU Sync “consistently slashes KV transfer latency by over 90% and reduces Time to First Token (TTFT) by an average of 90% for heavyweight open-weight MoE models (235B–480B parameters)”. The results list gives 60% for the 235B model and 90% for the 480B model, and the post does not say how the two statements relate, so both are reported here as written. The New Stack summarised the claim with a qualifier, saying Google “claims it can cut KV-cache transfer latency and time to first token by more than 90% in some tests”.

A second pair of figures also differs. The Google post says TPU Sync reaches “up to 223 GB/s of aggregate D2H and H2D throughput per Ironwood (TPU v7x) host”, out of a 256 GB/s PCIe Gen5 ceiling. The README describes “approximately ~200 GB/s bidirectional PCIe bandwidth”. None of the cited sources explains the difference, and the Herald does not reconcile them.

Where it plugs in

The Google post names Inferact and RadixArk as maintainers of the vLLM-TorchTPU and SGLang-TorchTPU projects and says it co-engineered the vLLM integration with Inferact. The vLLM-TorchTPU repository’s disaggregated-serving guide lists two KV-cache transfer connectors. One, TpuRaidenConnector, is described as a “High-bandwidth PJRT DMA/RDMA engine powered by tpu-sync-torch”; the other is a host TCP connector for “functional testing and local disaggregation”. The guide still uses the earlier Raiden name and contains no TPU Sync performance figures.

The Herald previously summarised InfoQ’s report on TorchTPU, the PyTorch backend that the TPU Sync post names as its PyTorch route.

Caveats in the sources

The README says TPU Sync “is under active development and its APIs are evolving rapidly” and recommends pinning to a tagged release or to the verified commit in lkg.version. It lists Cloud TPU v7x and v6e as the hardware the project is “actively validated and optimized” for. It also states: “Until the first stable release is cut, only .dev nightlies exist”, so installs from PyPI need the --pre flag. The Herald’s check of the repository’s releases and tags lists through the GitHub API on November 10 returned no entries.

The New Stack says the software “isn’t production-ready for everyone just yet” and relays context from SemiAnalysis: that external TPU serving previously lacked infrastructure to separate prefill and decode workloads effectively, and that SemiAnalysis’s InferenceX data shows Nvidia’s GB200 and GB300 NVL72 systems ahead in parts of the disaggregated-serving performance curve. According to The New Stack, “a direct comparison with a fully tuned TPU v7 setup has yet to be published”. The Google post points readers to InferenceX for “more up-to-date performance results”.

What we don’t know

• None of the four cited sources reports a measurement of TPU Sync’s latency or throughput made by a party other than Google. The New Stack relays SemiAnalysis context but does not attribute any TPU Sync measurement to SemiAnalysis.
• The sources do not say when a first stable release or tagged version will be cut.
• The Google post says frontier AI organizations “including Meta, alongside major enterprises like Wayfair” are adopting its PyTorch-on-TPU serving stack and evaluating TPU Sync multi-tier architectures, but none of the cited sources describes results from those evaluations.