Cloudflare discloses KV cache leak that mixed GLM-5.3 users' data

Cloudflare has disclosed a tenant isolation failure in its Workers AI serving of Z.ai's GLM-5.3 model: between August 27 and September 9, 2026, 155 responses contained data from other customers' requests made to the same model shortly before theirs. Every affected prompt was longer than 681,280 tokens, and the whole incident traces back to one number being smaller than another — a KV-cache pool sized below the model's advertised context window.

Key facts

• 155 responses leaked cross-tenant data between Aug 27 and Sep 9, 2026, all to prompts longer than 681,280 tokens.
• The prefill KV pool held 681,280 tokens against a 1,048,576-token context window — a mismatch Cloudflare knew about but expected the engine to reject.
• GLM 5.3 was pulled offline within 40 minutes of the September 9 incident declaration and restored the same day.
• Fixes landed in SGLang v0.5.20 through v0.5.22, the last released Oct 19, 2026.
• No evidence of deliberate exploitation, per Cloudflare's review of request timing and volume.

What happened

The report came from Gadi Evron of Knostic, filed through Cloudflare's HackerOne program on September 8. Cloudflare reproduced the leak the next morning using two independent accounts and set GLM 5.3 to private within 40 minutes of declaring an incident. By 19:40 UTC the model was back in public service with a corrected configuration and a patched engine.

The root cause was two overlapping problems. First, a misconfiguration: Workers AI deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on the prefill GPUs only had room for 681,280 tokens. Cloudflare's own disclosure says it was aware of the mismatch before launch, but its long-context tests checked HTTP status codes and engine performance, not output coherence, so nothing failed loudly.

Second, a bug in SGLang, the open-source inference engine Cloudflare uses, in how it handled over-length requests in disaggregated prefill/decode mode. In that architecture, prefill GPUs process the prompt and produce the model's KV cache, then hand it over remote direct memory access via Mooncake to decode GPUs that generate the response. When a prompt exceeded the pool, SGLang internally rejected it and swapped in a one-token placeholder — but the disaggregated admission queues checked whether a request was finished, not whether it was rejected. The rejected request stayed live through the handoff, prefill sent one page of KV data, decode reserved pages for the full prompt and generated while attending over stale cache pages that still held other users' data. The client got HTTP 200, a coherent reply, and someone else's context mixed in.

Cloudflare verified the fix against the researcher's reproduction script and its own canary-marker tests before restoring the model.

Why it matters

This is a shared-inference incident, not a model incident. GLM 5.3 itself — the weights Z.ai publishes — did nothing wrong. The failure came from the plumbing around it: a memory pool that did not match a configuration value, in an engine that did not reject the mismatch safely. Anyone running multi-tenant inference on disaggregated SGLang is in the same failure class, and Cloudflare's own recommendation to operators is to measure KV pool against context length from the startup log after every change to weights, quantization, or parallelism, then upgrade to v0.5.22 as the backstop.

For builders choosing where to run models, the useful question is not "is this provider safe" but "how does this provider verify memory isolation between my requests and the next tenant's." That question applies to every hosted endpoint, and it is the same class of question our private inference pillar covers: single-tenant deployments remove the cross-tenant boundary entirely, and trusted-execution-environment approaches attack a related trust problem from a different direction — confidential AI inference is one such path.

The disclosure posture is also worth reading. Cloudflare reported via HackerOne on September 8, restored service on September 9, met SGLang maintainers at RadixArk on September 10, and published only on October 26 once upstream fixes had shipped. That is the coordinated-disclosure timeline working as intended, and the SGLang team's own post describes the upstream side of the same work.

Background

Disaggregated prefill/decode is becoming the default serving architecture for very long-context models because it lets each phase scale independently. The trade is that a KV cache page gets touched by more code paths — prefill, transfer, decode, offload — and the engine must track who owns each page across nodes. Cloudflare's post catalogs a family of these bugs: pages freed while still in use, races between aborts and in-flight RDMA writes, stale confirmations. The incident bug was the deterministic version of the same species — a rejected request sailed through the handoff every time, not by timing luck.

Nothing here changes what we said about GLM 5.3 itself when Anthropic's red-team report on open weights landed: the weights and the serving stack are separate trust surfaces. The steady-state fix upstream is an engine that verifies, at read time, that a request owns every page it attends over. Cloudflare prototyped page-level tags this summer at under 1% throughput and tail latency cost, and SGLang's per-transfer checksum flag (--disaggregation-enable-kv-checksum) is a first step in that direction, though it does not yet cover every model architecture.

What's next

SGLang v0.5.22 (Oct 19) carries the admission fix and most of the hardening; the per-transfer checksum cleanup and the zero-on-release draft proposal (PR #46435) are still in flight. Cloudflare and the SGLang team are working to extend checksum and canary coverage to GLM 5.3, GLM 5.3 Flash, and DeepSeek V4, and Cloudflare is starting a formal-verification effort to model the RDMA KV transfer protocol itself. Operators running disaggregated SGLang: upgrade, then measure your pool.