Sparse attention can be simply understood as: instead of having each token look at all tokens, selectively look at only a part of them. This term comes up repeatedly in long context and inference cost discussions because although standard full attention is strong, once the context is particularly long, the cost of computation and video memory rises quickly.
Why is "reading it all" getting more and more expensive?
In standard attention, the longer the text, the more relationships need to be calculated between each other. It's okay when it's a short text, but when it comes to long documents, long code, and multiple rounds of history, the pressure on resources will quickly rise. The idea of sparse attention is: since not all positions are equally important, don't count them all.
How sparse it is usually
- Only look at local neighbors and retain recent context.
- Give a small number of key locations global visibility, such as titles, anchors, or summaries.
- Follow the rules or learned patterns to choose the more noteworthy segments.
Why this concept is getting hotter
- The long context has changed from an "experimental selling point" to a real product demand.
- The larger the model and the longer the context, the more careful the inference cost will be.
- Inference models and agent tasks are more likely to lengthen links, and architecture layer optimization is even more important.
But sparse attention is not in vain. If you miss some content, you may miss out on long-distance dependencies and hidden associations. So the key is not "the sparser the better", but how to balance retaining important information and controlling costs. It is precisely because of this tension that when the long context is hot, the sparse attention will become hot again.
