Model pruning gives a large model a haircut: find the weights that sit near zero and barely affect the output, delete them or zero them out, and the model gets smaller and faster while losing only a little ability. It is often grouped with quantization and distillation as model compression, but the three cut in completely different places, so this entry starts with pruning itself.
What exactly gets cut
What fills a neural network is weights, the strength values on the connections between neurons. After training, those weights are unevenly distributed: a minority of connections carry most of the judgment, while a long tail sits so close to zero it is almost background noise. Pruning ranks connections by importance and removes that tail.
There are two styles. Unstructured pruning zeroes individual weights, leaving a file full of zeros; it saves space in theory, but ordinary hardware does not run faster on zeros without dedicated sparse-compute support. Structured pruning removes whole channels, attention heads, or even layers, leaving a regular shape that speeds up directly on ordinary GPUs and phone chips. In practice the second style is used far more, for exactly this reason.
Why the model survives losing so many connections
The key is over-parameterization: to make training work, models are usually given far more parameters than the task truly needs. Many connections are redundant from the start, cancelling each other out or activating on only a handful of examples. Removing them is like taking down scaffolding, not a load-bearing wall.
Pruning is not a one-shot cut, though. Hard deletion usually costs noticeable quality, so the standard routine is iterative: prune a little, fine-tune to recover, prune again. The lottery ticket hypothesis describes a related finding, that a large network hides a much smaller sub-network which, trained on its own, performs nearly as well. The ceiling on pruning depends on how much redundancy exists; a model already trained tightly has little slack to give.
How it differs from quantization and distillation
| Method | What it changes | Model structure | Typical gain |
|---|---|---|---|
| Pruning | Removes connections or whole channels | Smaller, sparser | Fewer parameters; structured pruning also speeds inference directly |
| Quantization | Lowers each value from 16-bit to 8-bit or 4-bit | Unchanged | Much lower memory use |
| Distillation | Trains a small model to imitate a large one | A new model | A standalone small model |
The three are often stacked: prune first, then quantize, then recover quality with distillation or fine-tuning. When an open model advertises a pruned variant, check whether it was pruned structurally or merely zeroed; the first genuinely runs faster, the second mostly just looks smaller.
Three common misunderstandings
First, pruning does not delete data; it touches internal parameters and has nothing to do with the training set. Second, more pruning is not always better: past the redundancy boundary, quality falls off a cliff, and what disappears first is usually niche knowledge and long-tail skills that common benchmarks never test. Third, memory saved by pruning does not mean room to train a bigger model; it mainly serves inference deployment, while training memory is dominated by optimizer states and activations, where pruning does not help.
Before pruning, ask two questions: is the deployment device truly constrained on compute and memory, and do you have data for recovery fine-tuning? If both answers are yes, proceed; if you only want a prettier file size, the quality risk is not worth it.
