Sparse Attention
Explains how sparse attention works, its main implementations, and the trade-offs: how models compute attention only over key tokens to cut compute and memory costs on long contexts, and when the approach pays off.
Sparse attention keeps large language models efficient on long contexts: instead of full attention over every token, compute concentrates on the positions that matter — via block selection, token-level selection, or KV sharing. This collection traces the idea's evolution and representative architectures such as HySparse, and lays out the real trade-offs in long-horizon agent tasks and million-token inference.
