Together Compiler launched on November 4, as Together AI handed inference tuning to a swarm of agents: they rewrite GPU kernels overnight, benchmark them repeatedly, and send only verified versions into production. The company says that on NVIDIA Blackwell chips, four open models compiled for 24 hours reached up to 3 times the throughput of the best existing open engines. What matters is a change in how inference deployments are tuned — from researchers hand-tuning for weeks to agents retuning continuously as traffic and hardware change.
How the agents rewrite kernels: only verified versions ship
Together calls the approach polymorphic inference, and Together Compiler is its first production part. Swarms of agents propose kernel changes across a model's whole forward pass, while a persistent engine hot-reloads each candidate, measuring speed and checking correctness against the stock engine; winning kernels go into a shared registry, so the next model or chip starts from what earlier runs learned instead of from scratch. When a model, hardware target, or workload changes, the optimization can run again instead of waiting for a researcher to find time. The team behind it previously built FlashAttention and ThunderKittens; this is that kernel research turned into a product.
The numbers impress — but check what they were measured against
The official page gives several figures: at the same per-user speed, four models on Blackwell delivered up to 2.6 times the tokens per GPU; GLM reached up to 3.7 times on AMD MI355X and MiniMax M3 up to 2.7 times on AWS Trainium2; Llama-3.2-1B beat an expert-written megakernel on H100 and B200. In a customer case, voice company Phonic said its median time to first speech fell by nearly half overnight. These are Together's own benchmark results, measured against the “best open engines” or “best public benchmarks” — not a universal speedup for every deployment; real gains depend on the model, traffic shape, and hardware combination. Compiled kernels are already on by default for some serverless models, including GLM 5.3, GLM 5.3 Flash, and DeepSeek V4.1 Flash, at the same price; dedicated endpoints require a limited-beta application, with self-serve optimization to follow. Together previously launched Together Link, which lets Claude Code and Codex switch to open models; taken together, the two moves push open models' usability from the access layer into the serving layer. For teams serving open models, the next thing to watch is whether self-serve optimization delivers similar gains at smaller scale, and whether cross-hardware stability holds up under real traffic.
