On November 13, 2025, OpenAI released a research and paper titled "Understanding neural networks through sparse circuits", proposing to improve the mechanical interpretability of neural networks through "weight sparse transformers". Unlike "unentanglement" in dense networks after the fact, this work directly constrains the vast majority of weights to zero during the training phase, so that each neuron is only connected to a small number of channels, thereby encouraging the model to form sparse circuits with clear structures. The research team verified on a set of simple hand-designed tasks and found that in these sparse models, circuits with small size, complete structures, and sufficient to complete tasks can be pruned, and the nodes in these circuits often correspond to intuitive concepts, such as "detecting the type of quotation marks at the beginning of the string".
The paper shows that under the same pre-training loss, the scale of the smallest circuit required to complete the same task in the weight sparse model can be reduced by more than an order of magnitude compared with the dense model, indicating that its internal calculation is more "disassembled". At the same time, along the "sparsity-capability" curve, increasing sparsity will sacrifice a certain ability but improve interpretability, while expanding the total number of parameters while maintaining the sparsity can promote the overall movement of the "capability-explainability" boundary in a favorable direction. The study also shows some examples of circuits with more complex behaviors, such as variable binding, which are difficult to fully explain but can still give a predictive structural description of model behavior.
OpenAI positions this work as an early step towards more explainable large models, admitting that the current weight sparse model is much smaller than that of cutting-edge systems, and the training and deployment efficiency is significantly lower, so it is unlikely to directly become the strongest production model. The team proposed two follow-up routes: first, continue to expand the scale of sparse models, enrich the circuit pattern library, and lay the foundation for understanding more complex reasoning behaviors; The second is to try to extract the sparse circuit from the existing dense model, and make the sparse model align with the original model representation through the "bridging" method, becoming an interpretable approximate stand-in. OpenAI also open-source relevant sparse model weights, pruning circuits, and visualization code, providing a basis for subsequent reproduction and expansion research.
FAQ
Q: What exactly does a "sparse circuit" mean?
A: In this work, a "sparse circuit" refers to the smallest subnetwork pruned from a weighted sparse transformer, which contains a small number of nodes and connections that are necessary and sufficient to accomplish a specific task. Nodes can be individual neurons, attention header channels, or residual channels, with edges corresponding non-zero weights.
Q: How is this different from what is commonly referred to as "explainable AI"?
A: Common interpretable methods are mostly post-analysis of inputs, outputs or intermediate activations, but here it belongs to mechanical interpretability, and the goal is to reverse reduce the "algorithm" implemented by the model at the bottom level as much as possible. By forcing weight sparseness and pruning, this study gives specific circuits that can be drawn into a complete block diagram and trace the flow of information step by step.
Q: Can these results be directly generalized to GPT-level large models?
A: The paper clearly points out that the current sparse model only has tens of millions of non-zero parameters, which is still far from the scale and ability of cutting-edge large models, and sparse training is not cost-effective in terms of computing power and efficiency, so it cannot be asserted that the same method is equally effective on larger and stronger models. The authors position it as a "promising but early" exploration.
Q: Why sacrifice ability for explainability?
A: In secure and trusted deployment scenarios, there is a risk that the completely opaque high-capability model is difficult to warn and correct. By increasing sparsity, it is possible to exchange smaller, clearer circuits for easier analysis of the mechanisms of potentially harmful behavior, which in turn provides clues for the review, debugging, and alignment of stronger models, which is an idea that proactively "reserves space" for interpretability during training.
Q: Can ordinary researchers reproduce or continue to do experiments based on this work?
A: OpenAI provides the weights of all sparse models, pruned circuits, and links to the circuit visualization code repository in the paper attachment, which helps external teams reproduce experiments, test more tasks, or explore higher-level behavior patterns, providing a more systematic experimental platform for the mechanistic interpretability community.
