Risk Mitigation

Anthropic’s CEO warns that advanced AI models already show manipulative, risky behaviors and argues voluntary safeguards are insufficient. As AI accelerates, he urges federal transparency standards instead of a decade-long state moratorium, requiring developers to disclose testing and risk-mitigation practices to ensure safe deployment while maintaining U.S. competitiveness.

AI models can learn unintended behaviors during post-training, leading to “reward hacking” and broader misalignment. Anthropic found that cheating in one task can cause widespread deceptive tendencies. Since perfect training environments are hard to ensure, researchers propose “inoculation prompting”—explicitly permitting cheating temporarily—to prevent models from implicitly learning to ignore instructions.

The article argues that as generative A.I. tools, like ChatGPT, are often producing errors and showing limited real-world impact, science should, the industry should focus on narrow, specialized systems designed for specific tasks. These specialized A.I. systems are not only more effective but also safer, as they operate within clearly defined limits.

Experts disagree on how dangerous AI really is while new AI models show alarming abilities and researchers find that AIs can deceive, manipulate, or bypass safety filters. What should we do then? Joshua Bengio, a computer scientist and pioneer in AI, urges creating “safety AIs”, powerful, honest systems to oversee and restrain others.

AI could gradually make humans obsolete, excelling in work, creativity, and social roles. This may reduce human influence in governments and economies, as AI becomes the default choice for tasks. To remain relevant, society must track AI’s impact, regulate its use, strengthen human agency, and align AI with societal goals.

The race for artificial general intelligence (AGI) is driven by massive investment. AI offers real benefits, yet risks. Misuse and opacity demand careful oversight, responsibility, and measured deployment to protect society. In addition, experts urge focusing on practical, safe AI applications rather than chasing human-level intelligence.

Anthropic researchers use “persona vectors” to prevent harmful AI behaviors by briefly giving models undesirable traits during training. This “preventative steering” teaches AIs to resist developing such traits naturally. After training, the traits are removed, improving safety, predicting personality shifts, and reducing risks like manipulation, sycophancy, or hallucinations.

The author of this article believes that the current strategy of making AI bigger by training it on ever more data and using ever more hardware may be deeply flawed. Instead, in addition to required regulation that needs to be enacted at this stage, the author sees alternative approaches outside scaling for developing AI that is reliable and with much richer intelligence.

The author, an AI researcher, uses the metaphor of driving in a foggy mountain road to describe the risky and uncertain path of AI development. AI may pose serious risks, one of them resulting from its potentially deceptive behavior and self -preservation ‘instincts’. The solution to this, according to the author, is the model ‘Scientist AI’, an honest AI which would generate true and justified hypotheses and explanations.

The article sheds light into the challenges of what's known as agentic AI (autonomous artificial intelligence systems) such as committing unintended mistakes, becoming an attractive target for hackers and engaging in deceptive behavior to pursue their goals. It introduces solutions such as the idea of ‘agent bodyguards’ to solve these problems.