Opening the "black box": scientists have learned to read neural network thoughts

The inner workings of neural networks remained a black box, even for their creators. A new research direction makes it possible to change this for the first time.

Why neural networks are black boxes

Language models are not programmed—they are trained. During the training process, the model itself forms billions of numerical parameters. No one wrote them manually, and no one can explain exactly what each one does.

Anthropic CEO Dario Amodei admitted that the company does not understand how the models it created work.

Classical explainable AI methods work from the outside: they change input data and observe changes in the outputs. However, they do not reveal what happens inside the model during computation.

How scientists look inside

Mechanistic interpretability studies neural networks from the inside out. Researchers analyze weights and activations to identify specific structures responsible for specific behaviors.

There are two key concepts in this field.

A feature is a pattern of activations within the model that corresponds to a specific concept. It is not a single neuron, but a specific combination of activations—a single neuron can simultaneously participate in dozens of unrelated concepts.

A circuit is a connected path of features that implements a specific computation. If features are the dictionary, then circuits are sentences composed from that dictionary.

What was found inside Claude

Anthropic researchers deciphered several specific circuits.

The Golden Gate feature. Inside Claude 3 Sonnet, researchers found an activation pattern that triggered every time information about a bridge was processed. When researchers artificially amplified this feature, the model began mentioning the Golden Gate in every response, regardless of the topic.

Researchers discovered how Claude Haiku decides where to insert line breaks during text formatting. The model tracks character counts through a six-dimensional activation space, using spiral geometric patterns. Different attention heads are responsible for different distance ranges. No one taught the model this scheme—it emerged on its own during training.

Anthropic deciphered the complete circuit: which neurons recognize digits, which attention heads move information to the correct positions, and how the model handles carry-over processing.

How the neural network discovered mathematics on its own

In 2022, researchers trained a small model on a modular arithmetic task. Initially, it simply memorized the correct answers for the training set—accuracy on new data was near zero. Then, thousands of training steps produced no visible changes. After that, within a few hundred iterations, accuracy on new data rose from zero to nearly 100%.

When researchers examined the trained model, they found that it had independently discovered a solution using trigonometric identities and Fourier transforms. The model represented each number using sine and cosine, computed their products, and used the cosine sum identity to derive the answer. There was no trigonometry in the architecture—the algorithm emerged on its own as the most efficient way to solve the task.

This is the first case where researchers fully understood how a transformer solves a real-world task from start to finish.

Where the boundary lies today

MIT Technology Review named mechanistic interpretability a breakthrough technology of 2026. Anthropic has set a goal: by 2027, to reliably identify most issues in model behavior using interpretability tools.

Meanwhile, the world’s best researchers, using the best tools available, have so far been able to fully explain only how Claude adds two-digit numbers, inserts line breaks, and matches rhymes. How the model reasons, writes code, and analyzes documents remains unknown.

The gap between what can be explained and what modern models can do is the main challenge for the entire field.

Source: anthropic.com