Priming is an architecture-conversion method that an Amazon Web Services team unveiled on October 12, 2026, in an official Amazon Science post, with the accompanying paper released the same day. The problem it attacks is concrete: testing a new hybrid architecture used to mean pretraining a model from scratch, at a cost of trillions of tokens and hundreds of thousands of GPU-hours. With priming, developers start from the weights of an already trained Transformer, replace some of its attention layers with state space layers, and cut the pretraining cost by more than 99%; the whole pipeline consumes less than 0.5% of the source model's pretraining token budget.
The Transformer's memory tax: every layer remembers everything
A Transformer's attention works like verbatim note-taking: each new token looks back at every earlier token, and the cache grows linearly with context length, what the authors call eidetic memory. But not every layer does long-range retrieval. Some layers only track nearby words and local structure, and for those a fixed-size summary of the past, a fading memory, is enough. Hybrid architectures split the work accordingly: attention stays where global recall matters, and state space layers compress history into a fixed-size state elsewhere. The obstacle was never the idea but the validation bill: every new ratio or state space variant demanded a fresh pretraining run that almost nobody could afford.
Three steps: pick layers, align them, then fine-tune
Priming proceeds in three stages. First, layer selection: each layer is temporarily replaced with sliding-window attention, and the layers whose performance degrades least are flagged as local workers and prime conversion candidates. Second, weight transfer and alignment: the query, key and value projections of the original attention layer initialize the state space layer, followed by a short distillation that teaches the new layer to mimic the one it replaces. Third, end-to-end adaptation: the full hybrid model is fine-tuned on next-token prediction so old and new layers learn to work together. The team reports the pipeline works across Transformer families including Qwen, Llama and Mistral, on dense and mixture-of-experts models, and at scales from eight billion parameters to 32 billion and beyond.
Three state space layers, one controlled race
Under identical data, budget and hybrid ratio, the ranking of the three state space families was stable: Gated KalmaNet (GKA) first, Gated DeltaNet second, Mamba-2 third, with the gap widening as context grows. A 32-billion-parameter GKA hybrid matched its source Transformer on reasoning benchmarks to within 1%, halved the KV cache at inference, decoded up to 2.06 times faster at 128K context, and reached target accuracy in as little as two-thirds of the wall-clock time. A counterintuitive result followed: every state space hybrid beat its sliding-window-attention counterpart, even when the window version received two to four times the memory budget. GKA also offers a knob the others lack: its iteration count can be adjusted at serving time, so dropping from 30 to 10 iterations costs roughly 1% accuracy on AIME 2025 while decoding 10% faster, letting one trained model serve several latency tiers.
The model zoo is open; the real shift is the cost of asking architecture questions
AWS also released a zoo of primed hybrid models at eight-billion and 32-billion parameter scales for long-context reasoning and instruction following, together with training code, optimized kernels and a vLLM serving plugin, all under the Apache 2.0 license. Priming matters less as one more hybrid design than as a change in what architecture exploration costs: when trial and error drops from a full pretraining run to a fine-tuning budget, questions like which layers can be replaced, how many, and with which state space layer can finally be answered systematically at large scale. For cost-sensitive deployments the payoff is tangible, because KV cache savings translate directly into how many long sessions fit on one accelerator. The caveat is equally clear: all evidence so far comes from AWS's own paper and experiments, the source models are mainly Qwen3, and results on other families and smaller scales still await outside replication.
