Autoregression is the basic method a large model uses to generate text: it predicts the most likely next token from the tokens that have already appeared, appends that new token to the end of the input, predicts the next one, and repeats the cycle until it outputs an end marker or reaches a length limit. When you watch a chat answer pop out word by word, you are seeing the outward form of this step-by-step decoding, not an extra effect layered on top.
Pushing forward one token at a time
A token can be roughly understood as a unit of text in the model's eyes: it may be a character, part of a word, or a combination involving punctuation. At each step the model faces all the context so far, and it has only one judgment to make: what is the most likely next token? Once that token is chosen, the context grows by that amount, and the next judgment builds on the longer new context. Change even one word earlier on and the later direction can shift, because every step rests on all the choices made before it.
Training can check the full text, generation has to squeeze forward
Training and generation are different things. During training the complete original text is already there, so the model can see a whole sentence at once and practise, in parallel, what the next token should be at each position, like studying with the answer in view. During generation there is no ready-made continuation to look at: the model can only generate one token, attach it, and generate the next, so the work has to proceed serially. This is the main reason long answers are slow. It is not simply that the machine is not fast enough; structurally, this method cannot compute later tokens in advance, because it must wait for earlier results to land first.
Input can be read at once, output must come step by step
When you ask a question, a long passage you type in can be read in one go and processed in parallel, so a long prompt usually does not multiply the wait by its length. An answer is different. However many tokens the output contains, that is how many decoding steps it needs, and each step requires a fresh round of computation. The longer the answer, the more steps there are, and the longer it naturally takes. This also explains why, for the same question, a short reply returns quickly while a long one makes you wait while it writes itself out: the bottleneck is mostly on the output side, not the input side.
Not every generative model works this way, and three misunderstandings need clearing up
Autoregression is only the choice made by today's mainstream chat models, not the only route to generating content. Diffusion-based text models start from a blurred state, gradually remove noise and revise the whole, while masked models work like fill-in-the-blank exercises, hiding part of the content first and then completing it; neither relies on squeezing tokens out one by one from left to right. There are three common misunderstandings. The first is to think the model plans the whole passage before typing it out; in reality there is no global finished draft, later text is always influenced by earlier text, which is one reason it can drift further off course the longer it writes. The second is to think word-by-word output is a typing effect created on purpose; in reality, streaming display in the interface simply shows the decoding process in sync. The third is to treat sampling as a different generation method in itself; parameters such as temperature only affect which candidate token is picked at each step, and they do not change the autoregressive loop itself.
