Sparse attention

What is Sparse Attention? Why long context and inference cost issues always talk about it

Sparse attention can be simply understood as: instead of having each token look at all tokens, selectively look at only a part of them. This term come...