Positional encoding is an extra ordering signal attached to every word in a sentence, so the model knows which token comes first and which follows. Self-attention compares all words with each other but has no built-in sense of order: without position information, "the dog bites the man" and "the man bites the dog" would look like the same bag of words to it. Positional encoding fills exactly that gap.
Why self-attention cannot see order
Attention computes a similarity score for every pair of words, regardless of where they sit. Shuffle the input and the computation barely notices. Language, however, lives on order, so position has to enter the token vectors as a signal of its own before the model starts reasoning.
Three classic approaches
First, the sinusoidal encoding from the original Transformer paper: fixed wave patterns at different frequencies are added to each token vector. It needs no training and extends to longer texts, but it is rigid. Second, learned positions: every slot gets a trainable vector. Flexible, yet it stops at the longest length seen in training. Third, RoPE, the rotary encoding used by many current models: it rotates parts of the vector according to position, so attention naturally perceives relative distances. RoPE handles long contexts better and has become the de facto standard.
Absolute and relative position are different things
Absolute encoding says "you are word number 7"; relative encoding says "you are three words behind the reference word". For language, distance usually matters more than the house number, which is why modern schemes favour relative information. It also explains why models often turn fuzzy on texts far beyond their training length: they run out of familiar distances, not out of memory.
Three common misunderstandings
Positional encoding is not understanding: it supplies order, not meaning. It is not memory either: when a model loses an early detail, attention and the context budget are usually to blame, not the position scheme alone. And a longer context window does not automatically mean better reading at a distance: tricks such as scaling the RoPE base stretch the range but dilute resolution for nearby positions.
So when judging whether a model truly reads long documents well, ask not only how big its window is, but how it encodes positions and resolves distances. Positional encoding is the quiet foundation of that ability.
