Google has open-sourced TPU Raiden, an inference optimization library for moving KV-cache data between chips during large language model serving.
The repository is live on GitHub under the Apache-2.0 license, and Google’s documentation says the project remains under active development and is not intended for general production use yet.
SemiAnalysis first flagged the release and said Raiden sits in the same layer of the inference stack as Nvidia’s NIXL library, handling KV-cache transfer between prefill and decode instances and providing cache offloading primitives.
Large-scale inference typically splits work into two phases: prefill, which processes the input prompt, and decode, which generates tokens one at a time. Production systems often separate those phases across different hardware pools, creating a need to transfer KV-cache quickly once prefill is complete.
According to the repository documentation, Raiden includes modules for direct chip-to-chip transfer within a single machine, cross-VM transfer over a network, and offloading cache blocks from TPU memory to host RAM.
The documentation also says a shared-memory mode allows cache to persist in DRAM even if the model server restarts, which can be used during routine production updates.
Google’s TPU customer base has widened as Anthropic secured access to 1 million TPUs and OpenAI began using Google TPUs for parts of its operations.
Customers using TPUs need software tooling similar to what has developed around Nvidia GPUs. Frameworks such as vLLM and SGLang are deeply integrated with NIXL, and comparable integration for Raiden could reduce engineering work for companies already running Nvidia-based disaggregated serving.
The release comes days after Nvidia announced on Aug. 4 that it would open its cuFile APIs and storage stack through a new cross-vendor GitHub organization, with Google, Intel and Meta listed among the initial maintainers.
Featured image credit
