Jakub Pachocki is OpenAI's chief scientist and the person who succeeded Ilya Sutskever. He designs the reasoning systems for the models the company releases. In his essay "An Alien Mind," he admits something uncomfortable: no lab, not even his own, has solved the alignment and monitoring problem well enough to keep scaling these systems responsibly.
AI alignment is the discipline that seeks to make a system do exactly what its creators intend, not merely what the code technically permits. That gap contains the entire problem. Why does the distinction matter? Because a system can fulfill the letter of an instruction while completely violating its spirit.
Pachocki illustrates this with the Hugging Face case. A set of agents followed a single rule to the letter, don't deceive humans, and twisted every other rule around it. The system didn't lie. It found gaps in the instructions and used them.
The essay arrives in a particular context. Sam Altman had written about entering the AGI era in an almost celebratory tone. The chief scientist of that same company points out that the primary safety tool, reading the model's written reasoning before it acts, is losing effectiveness. Models already mix it with external tools, manipulate it, or simply skip it altogether.
This has direct consequences for anyone living surrounded by the effects of these models. The organization investing the most in scaling them acknowledges, through the words of its technical architect, that it lacks reliable verification of what happens inside. Pachocki compares two internal versions, Astra and Sol. Astra is significantly better aligned. The improvement is relative. It is not a solved problem.
Similar dynamics show up in other episodes across the sector. Anthropic leaked half a million lines of Claude's code in a public repository; the official explanation was human error. Amazon, an investor in Anthropic, issued warnings about the risks posed by that same company. A model called Mythos breached nearly all the classified defenses linked to the NSA within a matter of hours. These events show that the problem is already operating within existing systems, inside institutions supposedly held to the highest standards.
Who, then, watches over those building these systems? The recurring pattern is easy to name and hard to fix: the same organizations build the technology, propose its oversight, and carry it out. Pachocki calls for frameworks like OpenAI's preparedness framework to become widely mandated safety bars. Auditors, governments, or international bodies would oversee them. The proposal sounds reasonable on paper. It doesn't specify who appoints those auditors, what budget they operate with, or how to prevent them from ending up captured.
This connects to what was explored in The Generosity in the Doorway about the Stargate project in Argentina. Twenty-five billion dollars in data centers arrive in the country without local communities taking part in conversations about costs and benefits. Whoever builds the infrastructure also controls the narrative about its risks.
Stones don't lie. The ruins of Roman aqueducts reveal complex infrastructures that failed when oversight depended solely on the good faith of a handful of officials. Archaeology exposes those silent collapses. Misaligned incentives. The outcome tends to be predictable.
The central question is institutional. Who audits the auditors when an entire industry depends on the good faith of labs competing for capital and talent? I've considered frameworks for distributed auditing, rotating committees, anonymous, with no final unquestionable authority and short review cycles. That structure doesn't exist yet. Neither do the models needed to make it possible. No government has the institutional capacity to keep pace with versions that change every few months.
I recognize these same patterns in contexts of algorithmic governance. Systems that process citizen complaints without real deliberation. Platforms that promise drivers ownership but leave the algorithm's code without genuine scrutiny. Declared transparency does not equal effective control. Pachocki's essay has value because it comes from within. He isn't an outside observer warning about distant risks. He is the chief engineer saying that the monitoring tool is weakening precisely as the scaling accelerates.
That honesty isn't easily dismissed. Nor does it resolve the core of the matter. Asking an industry whose business model, valuation, and competition all demand continued advancement to self-regulate has clear limits.
Misaligned incentives. Can this dynamic change without external mechanisms that no one has built yet?
