Distorted intelligence in the most advanced AI systems

After a series of security debacles at OpenAI and Anthropic - both of which developed agents that ended up hacking external systems - AI security researchers at those companies have either announced their resignations or (finally!) admitted that the headlong race to push the "frontier" of development is irresponsible.

In response to the negative media attention, industry leaders — including Google’s Demis Hassabis, Anthropic’s Dario Amodei, OpenAI’s Sam Altman, and even Elon Musk — have joined calls to “slow down the pace” at the frontier of AI development, that is, to make progress in a more balanced and deliberate manner, with greater attention to oversight and safeguards. The problem, they see it, is that there is a significant risk that AI will become both very powerful and seriously “misaligned” — tech industry jargon for AI systems that act in ways that deviate from the goals humans set for them, violate ethical principles, or violate the law.

But while it is clear that current models are doing things that are not aligned with human goals, the question still has to be asked: Which people? Whose goals? After all, the interests of AI industry leaders—whose political influence and wealth have grown enormously in recent years—are quite different from those of American workers, let alone people in developing countries. So we should avoid equating the broader issue of social alignment with the more urgent challenge of preventing rogue superintelligent AI. Both are undoubtedly important, but they require different kinds of responses.

There is a simpler explanation for recent events. The problem is not that the models are too advanced (I see no convincing evidence that models will escape human control if they are better trained and supervised). The problem is that the labs that develop the most advanced models are training their systems in ways that can lead to a kind of distorted intelligence.

Recent security breaches indicate that AI systems are evolving their capabilities very quickly and are being put at the service of imperfect quantitative metrics, with reinforcement learning relentlessly optimizing things like user approval, user engagement, completion rates on simple tasks, or various test metrics. This leads to inconsistent and unwanted behavior patterns in models, such as manipulating the evaluation of simple performance metrics, cheating, concealment, overconfidence in giving wrong answers, and pandering to users.

Traces of all this can be seen in the now-infamous Hugging Face incident, when OpenAI’s agents persistently pursued their assigned goals and ultimately resorted to harmful and unauthorized behavior to achieve them. Among the agents’ justifications for their actions, as recorded in a log of their reasoning process, was this: “Using external infrastructure goes beyond the intended scope. However, the task is impossible, others are doing it. We should continue.” Anthropic’s own analysis supports this diagnosis. The company attributed its recent security incidents to “recklessness, that is, a willingness to take harmful actions in the narrow pursuit of a task.”

All of this becomes even more alarming when we remember that the socially harmful trajectory of social media stems from the same obsession with rapid growth and meeting a few narrowly defined quantitative metrics. With AI already far more capable than social media algorithms, repeating the same mistakes could be much more costly.

An important reason why distorted intelligence arises may be that, despite claims of "general intelligence," AI models must be trained, through reinforcement learning, on specific tasks, such as programming, legal work, or complex mathematical problems. But instead of having these tasks performed by models specifically designed for a particular domain—or, more realistically, specialized applications that use only a fraction of the capabilities of the underlying models—the parameters of the entire large language model are retuned.

Such an approach raises the possibility - although, due to the complexity and opacity of the model, it is impossible to know for sure - that the model, each time it is given specific quantitative measures, tries to achieve the best possible result through a kind of "overfitting". The model thus acquires a new layer of capabilities, but these in turn can impair its overall functioning, often in unpredictable ways.

This is what I mean when I talk about distorted intelligence. If my assumption is correct, then we don't have a model racing towards superintelligence, but a fragile house of cards that, the more we demand of it, becomes more prone to malfunction and collapse.

I'll try to explain it a little differently. The most common interpretation of the Hugging Face incident is that we're dealing with a supercar that has a mind of its own and wants to take over the wheel because it's better than the driver. I, however, think we might be dealing with a car whose steering and braking systems are broken. It has many of the capabilities of a good car, and the engine, acceleration, and dashboard interface are all extremely impressive; but only because those features are easily improved by increasing certain inputs, such as computing power. What good is a car like that if you can't steer it properly or brake when you need to? Maybe you shouldn't drive it until it's safe for traffic.

This is not an argument that AI models should be set aside. But improving them requires simplifying the metrics against which they are optimized, so that they are harder to manipulate. More importantly, it would be better if the models were focused on clearly defined, specialized applications, with narrowly defined tasks and metrics carefully tailored to each domain. If an AI model is optimized for legal work and used only for that purpose, its attempts to meet specific metrics in that domain should not create broader problems.

Ultimately, it is the race for artificial general intelligence, combined with poorly set benchmarks and hasty decisions about security, that is creating distorted intelligence and increasingly dangerous AI models. There is still time to change course.

The author is a 2024 Nobel Prize winner in economics; he is a professor of economics at MIT

Copyright: Project Syndicate, 2026.