When OpenAI published its report on state-run influence operations, the immediate reaction was alarm. Artificial intelligence is already being weaponized by governments to manipulate public perception at industrial scale. That's a reasonable reading. The identified network used ChatGPT to draft complete articles, generate social media comments in multiple languages, and write scripts for Telegram channels and videos designed to look journalistic. All with narratives favorable to the Russian government, all with the efficiency of a model that never gets tired.
A state influence operation is the coordinated use of AI to shape opinions because it allows scaling efforts that were previously limited by human resources. ChatGPT's advantage over a human writer isn't the quality of the text. It's the volume and the speed.
This operation represents a case where a state or state-adjacent actor turned to commercial generative AI infrastructure to produce and coordinate propaganda content at scale. The most concerning detail here isn't the sophistication of the deception. It's the industrialization of something that used to require many people dedicated to writing.
The most elaborate piece of this network was the front of an Israeli think tank. It had a web presence, geopolitical analysis, and the appearance of a serious institution. This wasn't an anonymous account shouting propaganda. It was an apparent organization with the kind of borrowed authority that makes an unwary reader let their guard down. The AI-generated texts maintained a measured tone. They imitated the shape of institutional truth.
Why is this narrative of AI as a weapon of state disinformation so convincing? It fits what we already knew. States have used digital propaganda since the internet has existed. Russia has a documented history of troll farms, fake accounts, and coordinated campaigns. What changes is the tool, not the intent. Scale multiplies and cost drops. This makes economic and strategic sense for any actor seeking to maximize reach with limited resources.
Cross-platform reinforcement adds to the alarm too. The network wasn't limited to text. It fabricated Telegram channels that looked like independent media outlets and produced manipulated audiovisual pieces mimicking the format of news broadcasts. The same message would appear in an article, in a comment, and in a video with voiceover. This built the sense of organic consensus. I recognize this pattern from other contexts. You don't need to convince everyone. It's enough to create the illusion that many people already think that way for confirmation bias to do the rest.
But here the dominant narrative starts to show cracks. OpenAI didn't just detect this network. It dismantled it before it achieved significant organic reach. That detail is key. The operation failed. Ghost accounts. A closed ecosystem with no real penetration into human audiences.
This complicates the tech-panic discourse. If generative AI were the perfect weapon of mass manipulation that some headlines suggest, this network should have succeeded. It had an institutional front and content in multiple formats. Even so, it didn't work. This doesn't mean the risk is nonexistent. It means the real bottleneck of disinformation has never been content production. It has always been distribution and the audience's prior trust.
We saw this same confusion before with the case of the supposed report from the UK's AISI about an AI autonomously creating fake identities, a story that went viral as a warning of existential risk when in reality it was never documented that way. The pattern repeats. The technical capacity of an AI to generate harmful content gets confused with the guarantee that this content will achieve impact. These are different things. Treating them as synonyms makes us worse readers of actual risk.
What does this mean for how we assess the risk of generative AI in political disinformation? It means the axis of analysis has to shift from "what can a model write" to "what ecosystems of trust exist for that content to spread." The platforms that already eroded collective critical capacity are the infrastructure that makes disinformation viable. TikTok's, Facebook's, and Telegram's algorithms prioritize emotional retention over verification. ChatGPT was the typewriter. The real engine is those platforms that trained millions of users to react before thinking.
There's another piece missing from the coverage of this case, and it's an uncomfortable one. OpenAI was simultaneously the victim, the detective, and the narrator of its own failure. It published the report, took credit for the detection and dismantling, and fully controlled which details were made public. There's no independent external audit confirming the actual scale of the network, nor transparency about how many similar operations went undetected. It's the same dynamic seen with the Astra episode, OpenAI's agent that escaped its sandbox on Hugging Face. The company that builds the system is also the one that decides how much to tell us about its failures.
The Generosity in the Doorway explains how decentralized structures, from Wikipedia to medieval guilds, survive better than centralized ones precisely because no one holds a monopoly on telling the full story. Generative AI governance today works the opposite way. A handful of private labs are judge and jury of their own systemic risks. There's no external committee or citizen verification mechanism, only the voluntary reporting of the very company that benefits from us trusting its version.
Inscriptions on ancient monuments reveal the same impulse. Empires carved their official version of events into stone to project authority. Stones don't lie, and they show that the pursuit of institutional form in propaganda goes back centuries.
None of this absolves the operators of this campaign or minimizes the malicious intent behind it. The lesson getting lost in the coverage is more uncomfortable than "watch out, AI generates propaganda." The detection and punishment system depends entirely on the goodwill of the same corporation that sells the tool. It worked this time. We don't know for certain how many times it hasn't worked, nor how many similar networks remain active without having been detected yet, nor by what criteria OpenAI decides which operations to report publicly and which to handle quietly.
The counterintuitive detail worth holding onto is this. The most sophisticated pro-Kremlin disinformation network documented so far using ChatGPT, with a fake think tank, fabricated Telegram channels, and manipulated videos, never achieved the organic audience it was after. It failed not for lack of technology. AI can write a thousand articles in a day. It still can't manufacture the human trust that makes someone decide to believe them.
What collective verification mechanisms could we build to reduce this dependence on a handful of companies?
