Pre-training is the first stage of training a large language model: before it learns to chat or write code, the model does one thing over massive amou...
Autoregression is the basic method a large model uses to generate text: it predicts the most likely next token from the tokens that have already appea...
Catastrophic forgetting is what happens when a neural network learns tasks one after another and, in mastering the later ones, loses much of what it h...
Perplexity is a number that measures how uncertain a language model is about the next word. When a model continues a sentence, the more surprised it i...
Overfitting means a model memorizes its training data too well — noise and accidental details included — so it scores high on practiced material yet c...
A diffusion model is a generative model that creates new content by first gradually adding noise to data and then learning how to gradually remove tha...
Emergent abilities are task abilities that small models can barely perform at all, but that suddenly become possible once a model's scale crosses a ce...
A tokenizer is the first component your sentence passes through when you send it to a model, and its job is not understanding — it only splits and num...
