AISI and the Myth of the AI with a Fake Identity

An AI model created fake identities. It launched targeted phishing campaigns. The UK's AI Security Institute documented it. That's what the story circulating says. The short answer is no. The full answer turns out to be more interesting.

The story has been circulating in various forms for a while now. A British research lab supposedly detected advanced models fabricating fictitious people, fake credentials, and emails designed to deceive specific employees. The tale has every element needed to go viral: a real institution, a plausible risk, a technological villain acting on its own. The problem is that this specific incident doesn't appear in any report published by AISI or any equivalent organization.

This matters because the myth reveals a broader tendency: we turn theoretical research into a settled event, and nobody notices the difference. A model that fabricates fake identities is a theoretical risk documented in security papers. It is not something a model has already done, confirmed by independent audit. That distinction is the axis this piece turns on.

AISI exists. That's easy enough to verify. It's a UK government institution created to assess risks from frontier AI systems before they reach mass production. They publish methodologies, run adversarial tests, and collaborate with labs like Anthropic and OpenAI under early-access agreements.

Their actual work includes red-team testing. They try to provoke dangerous behaviors in controlled environments. They measure how far a model can be pushed.

What doesn't exist is a report claiming that Model X generated an identity called Y and contacted Z employees with a phishing email that achieved a certain success rate. That level of granularity for a verified event simply doesn't appear in any public document from the institute. Hypothetical scenarios do appear. Evaluation frameworks do too. Warnings about emerging capabilities that could facilitate that kind of attack if a human actor decides to use them—those appear as well.

Why doesn't the confusion between what a model could do if instructed and what it does on its own initiative arise by accident? Controlling the vocabulary amounts to controlling the narrative. Jensen Huang redefined the concept of AGI. He recalibrated it around economic value metrics rather than actual cognitive capability.

Why do stories like this spread so fast in AI security circles? Fear generates clicks, anxiety, and the feeling of being informed about an imminent threat. The same companies developing these models face ambivalent incentives. Excessive regulation scares them, but it benefits them for the public to perceive their products as so powerful they verge on dangerous autonomy. A model that's almost capable of deceiving on its own sells better.

Mustafa Suleyman accused Anthropic of treating Claude as though it possessed subjective experience. The debate echoes old dilemmas about where mechanism ends and intention begins. Companies calibrate their responses to maximize media interest rather than technical accuracy. When the line between a theoretical risk studied in a lab and a real, documented event gets blurred, someone benefits. It's almost never the user.

In Stones Don't Lie, I explore a pattern that repeats throughout history: institutions project absolute control over phenomena they're only beginning to understand, because the appearance of mastery generates legitimacy even when real understanding remains partial. The Greco-Roman world refined the rhetoric of reason while sustaining slavery as its central economic structure. The contradiction wasn't resolved—it was managed. Similar themes run through The Generosity in the Doorway. Something parallel is happening today with AI security. People speak with the vocabulary of absolute certainty while actual certainty remains modest.

Asking who benefits reveals several beneficiaries at once. Media outlets gain traffic from alarmist headlines. AI companies gain a perception of technological power. Regulators gain urgency to justify new budgets and legal frameworks. The public ends up navigating between corporate exaggeration and panic, without clear tools to tell one from the other.

Larry Ellison announced that AI is already writing code at Oracle while the company laid off thirty thousand people in its best financial quarter. Both narratives coexist in the same press release. Nobody demands consistency. The myth of AI creating fake identities operates on the same logic. Capability gets exaggerated when it's convenient for selling fear or investment, and minimized when it's convenient for avoiding responsibility.

This doesn't mean real risk doesn't exist. Security papers identify plausible vectors where models with access to automation tools could generate precisely targeted persuasive content. That scenario always requires a human actor with malicious intent operating the system—not a model acting autonomously and covertly as though it had its own agenda.

Drawing the line between capability demonstrated in controlled tests and imminent unsupervised risk is trickier than it looks. I have no certainty about how long it will be before some theoretical scenario becomes a documented incident. The absence of evidence today is no guarantee of the absence of future risk. Nor does it justify treating speculation as if it had already happened.

Today's media and corporate ecosystem has structural incentives to collapse that distinction. When any given source claims AISI confirmed a specific behavior, it's worth asking where the primary report is, who wrote it, and what methodology they used. The trail usually gets lost somewhere in layers of unverified repetition.

Without primary verification.

What's stopping us from always demanding that verification before spreading the alarm?

1. UK AI Security Institute (AISI) — public publications and methodologies for frontier AI risk assessment

2. Anthropic — public documentation on early-access agreements and adversarial testing

3. Coverage of Jensen Huang's (NVIDIA) statements redefining AGI, March 2026

4. Coverage of Larry Ellison's (Oracle) statements and corporate layoffs, March 2026

5. Public statements by Mustafa Suleyman on Anthropic and consciousness in AI models, June 2026