Together Compiler rewrites serving code for open models on the fly

Together AI shipped Together Compiler, a system that uses teams of AI agents to continuously rewrite and re-tune the code that serves open models. The company claims up to 2.6x higher tokens-per-second against the best open engine configurations, on hardware and workloads that never stop shifting. Every number is the vendor's own.

Key facts

• Up to 2.6x higher tokens-per-second claimed over the best-reported open-source engine configurations on Kimi K3, GLM 5.3, MiniMax M3.1-Flash, and DeepSeek v4.1-Flash.
• 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash at batch size 1, which the company calls a state-of-the-art on SemiAnalysis' InferenceX low-latency benchmark.
• 252x higher experiment throughput than baseline agentic optimization systems, driven by "persistent engines" that hot-reload candidate code in 9.0 seconds instead of a 467-second cold start.
• 1.7x higher TPS over hand-tuned megakernel and CUDA graph baselines on Llama 1B at batch size 1.
• Together says it serves "many trillions of tokens per day" across thousands of customers, the traffic TKCC reportedly optimizes in production.

What happened

The blog post, published November 4 with authors Reyna Abhyankar, Dylan Lim, and Simran Arora, argues that hand-tuning inference code has stopped scaling. The company counts thousands of serving regimes spanning new model architectures, hardware platforms, and customer traffic shapes, and points out that none of the top-10 open-source models on OpenRouter share an architecture. Serving code tuned last month lags the next checkpoint release.

TKCC's answer is a loop. Agents generate candidate optimizations for an inference engine, communicate over a shared message board, and test them against a tier of "persistent engines": fast ones that load a few layers for quick iteration, and slower ones that load many layers to verify a win holds on realistic data distributions. The startup math is the striking part of the company's own tables: a conventional engine relaunch costs 467 seconds across weight loading, KV-cache sizing, autotuning, and warm-up, while the TKCC hot-reload path reports 9.0 seconds to serving. That difference is what lets agents run orders of magnitude more experiments per hour.

The company claims the loop generalizes past language models, reporting speedups on open-source codebases for weather modeling, partial differential equations, time-series, and biology workloads, where it says TKCC beat Claude Code running with a goal directive on three of four models tried. Together brands the broader program "Polymorphic Inference": serving that recompiles itself as traffic drifts, the way financial markets reprice continuously.

All of this is Together's own reporting. The company published a linked technical paper but no independent benchmark run has appeared yet. The figures worth checking first are the 2.6x TPS claim and the InferenceX records, since SemiAnalysis publishes its benchmark methodology and configurations openly enough for a third party to reproduce the comparison.

Why it matters

Serving cost is the quiet input to every price a builder sees. When a provider squeezes more tokens per second out of the same H100 or B200, either the price per million tokens falls or the margin rises, and which of those happens is a competitive question. Our decomposition of the inference cost curve tracks how serving efficiency compounds across the industry; an agent-driven compiler that re-optimizes in hours rather than weeks is one mechanism behind that curve.

For open-model buyers specifically, the system targets the maintenance tax of self-hosting. Anyone who has run a fresh open-weight release through vLLM or SGLang knows the pattern: the reference config leaves performance on the table, and the tuning knowledge lives in a handful of expert heads and issue threads. If agent-driven compilation really matches weeks of expert tuning in a day, the gap between managed providers and a self-hosted stack widens on the serving side even as the models themselves converge.

The claim also matters for the competitive read on Together. Our comparisons of Together AI and its alternatives and Together versus Fireworks rank providers on throughput, price, and openness of the serving story. TKCC is Together's argument that its serving layer is a research asset, not a commodity wrapper. Whether it holds up is an empirical question the InferenceX leaderboard can answer.

Background

Together AI is a San Francisco company selling inference APIs, fine-tuning, and GPU clusters built around open-weight models, and it has historically been one of the first stops for new open releases. The InferenceX benchmark it cites comes from SemiAnalysis, an infrastructure research firm whose AgentX workloads simulate multi-turn agent traffic on frontier open models.

The idea of automating performance engineering has been building for two years: learned optimizers, compiler autotuning, and most recently agentic kernel work. What TKCC adds, per the post, is measurement throughput. Existing agentic optimization loops tear the engine down and rebuild it for every candidate, tens of minutes per cycle, so they starve on experiment volume. Persistent engines flip that: the engine stays up, hot-reloads a candidate forward pass, and keeps serving while the agents argue on the message board. The 467-to-9 second startup comparison is the table that sells the architecture.

There is a caveat in the fine print worth flagging. The Trainium2 results were run on a synthetic multi-turn dataset rather than AgentX, because, per the post's footnote, the implementation does not yet support higher concurrencies on the accelerator's limited memory. That is an honest disclosure and a useful reminder that the 2.6x figure does not apply uniformly across every hardware platform named in the launch.

What's next

The cleanest verification path is SemiAnalysis' InferenceX leaderboard: if the 1111 and 1205 TPS records hold under third-party configuration, the throughput claims gain standing. Watch also for whether competing providers (Fireworks, Cerebras, Groq) adopt agent-driven recompilation publicly, and whether Together details the "Polymorphic Inference" research questions its post gestures at, particularly how often production traffic actually forces recompiles. If TKCC's loop is real in production, the observable effect is price: serving costs per million tokens at Together relative to peers over the next two quarters.