Transformer Architecture
Why the Transformer became the default backbone of large models, what attention actually costs, and what newer routes like diffusion language models and single-stream designs try to change.
The Transformer became the default backbone because attention links any two positions directly and training parallelizes well. The cost is just as clear: compute grows with the square of sequence length, so long context gets expensive. Newer designs - diffusion language models, single-stream diffusion - attack the cost of generating one token at a time, or pull text and images into one generative path. Those trade-offs hold the key to new design.
