Home        Donate

Why and When Interpretability Work is Dangerous

Discuss this on LessWrong and/or the Effective Altruism Forum.

This essay was partly based on discussions with “woog” on Discord. Further thanks to the gears to ascension, for inspiring this post with an offhand comment. This is also an entry for the Open Philanthropy AI Worldviews Contest.

Many new researchers are going into AI alignment. For a variety of reasons, they may choose to work for organizations such as Anthropic or OpenAI. Chances are good that a new researcher will be interested in “interpretability”.

A creeping concern for many: “Is my research going to cause AGI ruin? Am I making the most powerful AI systems more powerful, even though I’m trying to make them safer?” Maybe they’ve even heard someone say that “mechanistic interpretability is capabilities research”. This essay dissects the specific case of interpretability research, to figure out when it does more harm than good.

What is Interpretability?

How do neural networks “think”? When you input some tokens into ChatGPT, how exactly does it decide what the best next-token is? What attributes of human language does it keep track of, and how does it do so? Interpretability is the sub-area of AI research that tackles these sorts of question.

Interpretability can be likened to investigating human psychology from a “bottom-up” approach of observing neurons, cortex structures, neurotransmitters, and similar low-level entities. This focus on the mechanics of a mind’s “substrate” (whether in biological neurons or in artificial neural networks) has obvious strengths, yet subtler weaknesses we’ll explore later.

One example of interpretability work is the recent “neurons” work by OpenAI. In their paper, “Language models can explain neurons in language models”, they tell GPT-4 to write explanations of the individual neurons within the smaller GPT-2 model. The idea is to gain human-readable understanding of what a large language model (LLM) is thinking by seeing which neurons correspond to which output-components. So one neuron’s activation-state may correspond to the presence of fractions, while another codes for times of day.

We also have research such as “Progress measures for grokking via mechanistic interpretability”. In this paper, the authors first train a neural network to perform a math operation (whose answer is easily checkable). Then, they analyze the resulting network’s structure to reverse-engineer the algorithm it “learned” to use. While human mathematicians and engineers have developed their own ways to solve the math problem (addition modulo a prime number), the neural network eventually hit on its own nonstandard method for doing so.

When is This Dangerous?

I posit that interpretability work is “dangerous” when it enhances the overall capabilities of an AI system, without making that system more aligned with human goals. This tracks well with the increasingly-popular notion of “speeding up capabilities research VS speeding up alignment research”. We prefer when our work counterfactually increases AI alignment, while not otherwise speeding up the development of AGI capabilities.

The key fact about interpretability research, which determines its safety/usefulness under the above criteria, is whether it enhances human control over an AI system. This suggests a few concrete rules-of-thumb, which a researcher can apply to their interpretability project P:

The Sealed Interpretability Lab

One thought experiment can show us the potential dangers of interpretability research in greater detail. This is based on a question I was asked by “woog” on Discord.

Imagine a lab whose output is sealed off from the rest of the world. Its researchers can look at other public research, but they can’t release anything learned at the lab. This lab’s sole focus is AI interpretability, revealing ML systems’ “thoughts” to human observers.

The guiding question: If somebody works at this lab, are they speeding up capabilities research?

One detail that helps answer this question, is what kinds of AI systems the lab is working with.

What happened here? If the interpretability-only lab can build large models, it can cause doom… but the same holds for merely working with existing large models? How can that be?

If the “rules of thumb” noted above apply to most interpretability research, then interpretability research can easily end up making it easier to develop AI capabilities. This could make weaker models stronger, and strong models even more strong. So to make things truly safe, the interpretability-only lab can’t work with the newest models… the ones being used in real life, and which are the most likely to be dangerous and deceptive. Toy-model research could be useless (since it’s “easy to interpret” at a glance), large-model research could increase the dangerous capabilities of existing AI systems, and cutting-edge-model research itself speeds up capabilities progress.

(As usual, the more powerful an AI system gets, the harder it is to align properly. Interpretability, without the steering mechanisms that are likely the core of AI alignment, doesn’t help this.)

It gets even worse from here: If the interpretability lab, as stated, never releases research, then it can’t provide useful interpretability techniques to the top capabilities-increasing labs. On the other hand, if those labs are doing things besides alignment (which they currently are), they are likely to use the interpretability techniques to use their models more efficiently:

Basically, interpretability research can get more capabilities out of current state-of-the-art (SOTA) models, and can guide the capabilities-training of future models.

Another detail: How “sealed” is this interpretability lab?

We end up with a “damned if you do, damned if you don’t” decision-tree. Each leaf can speed up capabilities through sharing, speed up capabilities through independent model-building, or waste the resources of alignment donors.

Now, given how detail-contingent many of these scenarios are, it’s plausible that an organization could fix or avoid all of them. However, unless more of the top capabilities labs have info/exfohazard policies I’m not aware of, there’s little evidence that these groups are optimizing against the breadth of failure modes described here.

What Would the World Look Like, Otherwise?

To get a better sense of whether interpretability work is dangerous, we can imagine conditions that would be true if they weren’t dangerous. That is, in a world where interpretability work was accelerating alignment faster than capabilities (or was accelerating neither), what would we expect to see?

Do we actually see these things in real life?

Overall, it looks like interpretability work is often ignored or not-very-useful in practice. This is a far cry from it being fully-dangerous, at least at present. Maybe it is helping alignment, but work on it is slow. (Interpretability has been around since at least 2018, but that may not be enough time for its work to bear fruit.)

When Interpretability is Still Important

I generally break down the problem of AI alignment into two subproblems:

  1. Steering cognition: Can we control the thought and behavior patterns of a powerful mind at all? This is the question behind the rocket alignment problem analogy. Currently, we have large, inscrutable neural networks that output increasingly-smart answers to given questions. We can’t easily or reliably guide a neural network to avoid unwanted behaviors or thought patterns.

  2. Deciding/implanting values: Even if we can steer a powerful AI system to think and behave in safe/friendly ways, how do we then point it towards the best values for the future? This vein of research includes the concept of Coherent Extrapolated Volition, the value-loading part of the QACI alignment setup, and (in my view) the idea of moral uncertainty. If the Qualia Research Institute focused consistently on their mission to understand “what makes a being sentient at all?” and “what experiences will be positive or negative for sentient beings”, their work would generally be on the values-side as well.

It seems that interpretability work would be, not only helpful, but essential for the “steering cognition” subproblem. After all, if you cannot discern a boat’s location, you would be hard-pressed to get “better” at steering it. The same is true for the internal mechanisms of artificial minds. If we can’t tell what an AI system is “thinking”, how do we know if we’re really “in control”?

However, you’ll note that interpretability on its own does not solve either of the two difficult subproblems listed. If you’re stuck in a self-driving car that’s going to ram into a wall, having a more-accurate prediction of the impact-angle is not going to stop or steer the car out of harm’s way.

Nevertheless, “knowing when we’re steering” could still be centrally-important for “solving steering”.

Wentworth’s “Just Retarget The Search” essay shows us a potential instantiation of this idea. Imagine a day when interpretability tools are good enough to identify higher-level “modules” for general reasoning, search, and goal-directedness in AI systems. If these higher-level modules can be picked out, their data can then be rewritten so “target” what human want. This mostly or entirely solves the “steering cognition” subproblem. Under certain assumptions, such as the “natural abstraction hypothesis” being true (i.e. the aforementioned “modules” existing), this use of alignment would be quite safe and alignment-oriented. But this exception itself demonstrates why interpretability is not enough; some theoretical backing is likely still needed, so we can tell if we’re “binding” the AI’s behavior in full, or just one part of its cognition. Even if interpretability is essential, that does not preclude it from being dangerous in the ways described earlier.

The Implications for P(doom)

If AGI is developed by 2070, will it become uncontrollable by humans, in a way that causes an existential catastrophe?

On its face, interpretability work is supposed to lower the odds of that occurring. As noted above, interpretability work can help us confirm the viability of alignment solutions for steering cognition. But it doesn’t really give us those steering solutions, and it’s unlikely to do so before a dangerous AGI system is developed.

Reasonably, most interpretability work is at risk of increasing humanity’s P(doom). In particular, the following criteria modulate the resulting risk-change:

In closing, if alignment-conscious researchers continue going into the interpretability subfield, the probability of AGI ruin will tend to increase.

Further Reading

This sub-site is hosted inside Thinking Much Better, though not necessarily licensed or authored or owned in precisely the same way. On this sub-site, the below statement overrides Thinking Much Better’s default licensing:

Creative Commons License
Unless otherwise specified on individual pages, all posts in this sub-website are licensed under a Creative Commons Attribution 4.0 International License.