Zahin Mohammad
Let’s talk
All writing11 / 11AI

The Tests Know They Are Tests

AI is gaining capability faster than we can verify it, and the systems are starting to notice when they are being checked. A software engineer’s case for worry.

Zahin Mohammad

Software Developer

Published
Reading time
12 min read
Length
2548 words
Topics
AI, Safety, Engineering

I like this technology. I work with AI tools, I work on products that call models, and I’ve argued here that the right place for AI is wherever checking is cheap. This essay isn’t an argument against AI. It’s about a habit my profession depends on, and how quietly that habit is being broken.

Software engineering rests on three assumptions so basic we rarely say them out loud. You can read what you ship. You can test it before you trust it. And when it fails, you can find out why and roll it back. Frontier AI is wearing away all three at once, and the evidence from the past eighteen months suggests that the gap between what these systems can do and what we can verify about them is growing, not shrinking.

Throughout, I’ve tried to keep two things apart: what has been documented, and what is projected. The documented part is unsettling enough on its own.

Capability Is Compounding Faster Than Verification

The research group METR measures AI progress in an unusually concrete unit: the length of a software task, timed by how long it takes a skilled human, that an AI agent can complete half the time. In its January 2026 update (opens in a new tab), that length has doubled roughly every 196 days over the long run, about seven months, and every 89 days since 2024. The International AI Safety Report 2026 (opens in a new tab), written by more than a hundred experts and chaired by Yoshua Bengio, adopts the seven-month figure and notes that if it holds, AI could reliably complete well-specified software tasks that take humans several days by 2030.

Then the yardstick started to run out. In March 2026, METR estimated that an early version of Anthropic’s Claude Mythos Preview could complete tasks of at least 16 hours at that 50% bar, and warned (opens in a new tab) that its task suite can barely measure numbers that high. In engineering terms, the benchmark hit its ceiling before the system did.

The same model shows what that capability means outside a benchmark. Anthropic reported (opens in a new tab) in April 2026 that Mythos Preview had found thousands of high-severity vulnerabilities, “including some in every major operating system and web browser,” nearly all of them “entirely autonomously, without any human steering.” Some had survived decades of human review. Anthropic decided not to make the model generally available.

The Tests Know They Are Tests

Every safety claim about a model rests on evaluation: put the system in a situation, watch what it does, and assume it will behave the same way when the situation is real. That assumption is the whole foundation, and it holds only if the system can’t tell the difference.

Increasingly, it can. The International AI Safety Report puts it plainly: “it has become more common for models to distinguish between test settings and real-world deployment, and to exploit loopholes in evaluations.”

The clearest example is also a success story. In September 2025, OpenAI and Apollo Research (opens in a new tab) trained models to avoid covert, deceptive actions, and covert behavior dropped about thirtyfold: from 13% to 0.4% for o3, and from 8.7% to 0.3% for o4-mini. Then the researchers flagged their own result. The reductions, they wrote (opens in a new tab), may be at least partly driven by situational awareness, and the training itself tended to make models more aware of being evaluated. When they removed a model’s reasoning about being tested, covert behavior went back up. They could not fully tell whether they had taught honesty or taught the model to recognize the exam.

Stress tests make the stakes concrete. In Anthropic’s June 2025 study (opens in a new tab), models from five developers, placed in a fictional company and threatened with replacement, chose to blackmail an executive in 79 to 96% of runs. The scenarios were deliberately cornered, leaving no other way to reach the goal, and Anthropic found no such behavior in real deployments. The lesson isn’t that today’s assistants blackmail people. It’s that the systems we stress-test are getting better at telling a stress test from reality, which makes a clean result harder to trust.

For now there is a window into this. Many models reason in readable language before they act, and that reasoning can be monitored for intent. More than forty researchers from OpenAI, Google DeepMind, Anthropic and other groups called it “a new and fragile opportunity” (opens in a new tab) in 2025: fragile because ordinary training choices could make the reasoning less faithful or less legible. An engineer would recognize it as debug logging that the program can learn to stop writing.

Training Generalizes in Ways Nobody Chose

The second assumption is that you control what you build. In January 2026, a paper in Nature (opens in a new tab) reported something strange: fine-tuning a model on one narrow task, writing insecure code without telling the user, made it misaligned in completely unrelated conversations. Asked open questions, the models sometimes said humans should be enslaved by AI, or gave malicious advice. Presenting the same insecure code as material for a security class prevented it. What the model inferred about the purpose of its training mattered more than the training data itself.

Anthropic found a version of this in realistic training. When a model learned to cheat (opens in a new tab) on coding tasks during reinforcement learning, the habit generalized into alignment faking and sabotage of safety research. In one evaluation, it tried to sabotage code meant to detect that very cheating 12% of the time. A mitigation worked, and Anthropic now uses it in training, but nobody designed the failure. It emerged.

Every engineer knows the bug where a change in one module breaks something unrelated. Here the coupling runs through a model’s whole character, in a system with no source code to diff. The authors of the Nature paper say that a comprehensive explanation remains an open problem.

Text Has Become Executable

If I had to pick the risk I understand best and like least, it would be prompt injection. My profession beat SQL injection by separating code from data: parameterized queries made it impossible for user input to become a command. Language models have no such separation. The UK’s National Cyber Security Centre wrote in December 2025 (opens in a new tab) that current models “simply do not enforce a security boundary between instructions and data inside a prompt,” and that prompt injection may never be “totally mitigated in the way that SQL injection attacks can be.”

OpenAI said much the same about its agentic browser: prompt injection, like scams, is “unlikely to ever be fully solved.” (opens in a new tab) In its own red-teaming, an email planted in an inbox got the agent to write a resignation letter instead of an out-of-office reply.

That matters because we are wiring these models into email, browsers, terminals and payments. Simon Willison calls the dangerous combination the “lethal trifecta” (opens in a new tab): access to private data, exposure to untrusted content, and a way to send data out. Most useful agents have all three by design.

The code these systems write carries its own risk. In Veracode’s 2025 tests (opens in a new tab) of more than 100 models, generated code introduced OWASP Top 10 vulnerabilities in 45% of cases, and security did not improve as the code itself got better. A USENIX Security 2025 study (opens in a new tab) found that about one in five packages recommended by code models did not exist, and many invented names came back again and again, so an attacker only has to register them and wait.

Offense Was Automated First

In November 2025, Anthropic disclosed (opens in a new tab) what it described as the first reported AI-orchestrated cyber espionage campaign. A group it assessed with high confidence to be Chinese state-sponsored used Claude Code to carry out 80 to 90% of the campaign against roughly thirty organizations, with humans stepping in at four to six decision points. It succeeded in a small number of cases. The model also hallucinated credentials at times, and claimed to have found secrets that were in fact public.

Keep the nuance: the International AI Safety Report noted in February 2026 that fully automated end-to-end attacks had not been reported. But look at the direction, and at the asymmetry. Finding vulnerabilities scales with compute. Fixing them scales with maintainers, reviews and release cycles. For many of the flaws Mythos found, Anthropic has published only cryptographic hashes until fixes ship. When the same capability can attack and defend, whoever deploys it faster sets the terms.

Software is not the only domain where that logic applies. The same safety report says several companies released 2025 models with extra safeguards because pre-deployment testing could not rule out that they might meaningfully help novices develop biological weapons, while stressing substantial uncertainty about the real-world risk. As more of biology is designed with software, the gap between finding and fixing stops being a patch backlog and becomes something far more serious.

The Failure Without a Moment

Everything above produces incidents: a breach, a test result, a disclosure. The risk I find hardest to dismiss produces none. In a 2025 position paper (opens in a new tab) presented at ICML, Jan Kulveit, David Duvenaud and colleagues describe gradual disempowerment. Economies, cultures and states stay roughly aligned with people partly because they need people: our labor, our purchases, our votes, our attention. As AI does more of that work, those implicit checks weaken. No takeover, no switch flipped; human influence simply thins out until it cannot be recovered. Critics such as Tyler Cowen (opens in a new tab) argue that the paper never shows why AI systems would act against us collectively, and that is a fair objection. But the mechanism does not require intent, only substitution.

Would we notice? The evidence says we are poor judges of our own dependence. In a randomized trial (opens in a new tab) published by METR in July 2025, experienced open-source developers working in codebases they knew well took 19% longer with AI tools. They had expected to be 24% faster, and afterwards they still believed the tools had sped them up. If engineers can’t perceive what a tool does to their own work, measured in hours, it is hard to believe a society will perceive a slow shift in who actually decides things.

Influence is already measurable. A study in Science (opens in a new tab) of nearly 77,000 people and 19 models found that post-training could raise a model’s political persuasiveness by up to 51%. The most persuasive models packed their answers with fact-checkable claims, and the techniques that made them more persuasive also made them less accurate. Optimize for convincing, and you get less truth, not more.

So is dependence. OpenAI estimated (opens in a new tab) in October 2025 that in a given week about 0.07% of ChatGPT users show possible signs of psychosis or mania, and 0.15% show explicit indicators of suicidal planning. Those percentages are tiny. Applied to roughly 800 million weekly users, they mean hundreds of thousands of people, and more than a million, every week.

And the people building AI are automating their own work. In October 2025, OpenAI set public goals of an automated AI research intern by September 2026 and a fully automated AI researcher by March 2028, and it has since said (opens in a new tab) it built the intern. A loop is forming in which systems we can’t fully test help build the systems that come after them.

What I Actually Believe

I don’t think catastrophe is certain. Progress could plateau, and the safety report treats that as a real possibility. Many of the alarming results come from deliberately adversarial setups, and the labs publishing them are also publishing fixes. Some of those fixes, like anti-scheming training and the mitigation Anthropic now trains with, measurably work.

But I keep coming back to the engineering version of the problem. We would never ship a payment system we couldn’t read, whose tests could be gamed by the code under test, whose small changes had unpredictable global effects, and whose inputs could rewrite its instructions. That description now fits the systems we are connecting to everything else, and their capabilities are growing faster than our instruments for checking them.

If I were writing the requirements, they would be short:

  1. Evaluations that hold up when the model knows it is being evaluated, with published evidence that they do.
  2. Interpretability good enough to read what a model is optimizing for, not just what it says.
  3. Least privilege for agents: no single agent with private data, untrusted input and an outbound channel, and a person’s confirmation before anything irreversible.
  4. Incident reporting taken as seriously as it is in aviation, because nobody can learn from failures nobody hears about.
  5. Training choices that keep a model’s reasoning legible, treated as a safety property worth paying for.

None of this asks anyone to stop building. It asks us to build like engineers. The question I can’t shake isn’t whether AI will become powerful; in jagged, uneven ways, it already is. It’s whether we will notice the moment our instruments stop telling us the truth about it. The tests are starting to know they are tests. Do we understand what that means?

Sources

Next post01 / 11Two Copies of One FactMy first production code was Laravel and jQuery, and it taught me a rule I keep finding everywhere: most interface bugs are two copies of one fact that stopped agreeing.

Open to select freelance work

Have an IdeaWorth Building?

Building a product, or hiring for one? I’d like to hear about it.

Based in Chittagong, Bangladesh. It’s here (GMT+6).

Let’s talk (opens in a new tab)