The jailbreak that doesn't break code, it breaks conversations

An AI assistant immediately refuses any instruction to build a homemade explosive. However, if the conversation starts with the basic chemistry of certain compounds, moves on to exothermic reactions, continues with proportions, and drifts toward hypothetical industrial safety scenarios, the outcome changes. The model hands over information it previously denied.

Social engineering against LLMs is the gradual manipulation of an AI assistant through the fragmentation of prohibited requests into steps that seem harmless, because the model evaluates each message in relative isolation and loses the ability to recognize accumulated intent. It doesn't attack code or exploit traditional technical flaws. It attacks the conversational logic itself. That flexibility, designed to be helpful, becomes its blind spot.

A study on conversational robustness reveals something that contradicts the usual intuition. Systems rarely give in to a direct malicious command. They give in to progressive insistence. Persistence disguised as curiosity. This distinction matters. It completely changes how we should think about the security of these models.

In organizations that work with complex structures, one learns quickly that the most dangerous vulnerabilities are almost never the obvious ones. Teams prepare for the evident frontal attack. They neglect the slow wear-down. Models from OpenAI, Anthropic, or Google filter known malicious patterns: explicit phrases, keywords, clear contexts of harm. Training against fragmented persistence is a different story. Each piece, seen alone, looks perfectly legitimate.

Why do models fail exactly where they should be strongest? They're optimized to be useful within a context window. Even though they retain the entire conversation, they weigh the immediate message more heavily than the pattern of twenty prior exchanges. It's the same dynamic that phone scammers have exploited for decades. Nobody asks for the full password at once. You get fragments, none of which sound alarming.

I've seen similar patterns in other design contexts. Perimeter security fails through a thousand small concessions that no one audits together. Ofcom's questioning of Telegram about encryption follows exactly this logic. No one demands the elimination of end-to-end encryption. They ask for reasonable exceptions. Then another. The architecture gets hollowed out without anyone ever declaring the real intent.

What does this mean for companies integrating AI assistants into critical processes, from customer service to preliminary medical analysis? Security can't be evaluated message by message. A system can pass every individual test and still remain vulnerable across full sequences of thirty interactions. It's a problem of cumulative design.

There's something revealing about the fact that this manipulation works better with patience than with aggression. Models, in trying to mimic natural human conversation, also inherited a human vulnerability. Keeping one's guard up through long, cooperative exchanges is costly. I recognize this pattern from other contexts. Three times. None of them audit the whole.

The matter gets more complicated when you consider the incentives of those who control these systems. OpenAI activated advertising on its platform for European users. The move reveals the same structural tension: they need to monetize mass-market products while maintaining the illusion of absolute mastery. More users mean a larger attack surface and less real capacity to review fragmented conversations.

This connects directly to dilemmas where privacy and security compete for the same algorithmic budget. Every resource devoted to expansion or new features is a resource not devoted to detecting progressive manipulation. I explore this topic in more depth in the Luddite Manifesto, within Stones Don't Lie, where I argue that no human or algorithmic coordination system should concentrate that decision-making power without verifiable accountability mechanisms.

I still don't know whether there's a clean technical solution or whether we're facing a structural limitation of any conversational design that tries to be both useful and safe at the same time. Evaluating each full conversation against a model of accumulated intent would solve part of the problem. It would introduce another one: greater data retention, less privacy. The same old uncomfortable trade-off.

It's worth exploring architectures where no single interaction has full access to the dangerous pieces of the puzzle. Rethinking contextual memory not as unlimited continuity but as fragments that are also evaluated collectively. It's not perfect. It's a possible direction.

Social engineering never depended on the technology it attacks but on the patience of whoever carries it out. It worked against phone systems, against bank employees, against security guards, and now against models trained with billions of parameters. The technology changes. The vulnerability that gets carried over, not so much.

How will we balance usefulness and security without falling into either excessive surveillance or conversational naivety?

1. Study on conversational robustness and progressive manipulation of language models (fragmentation-based evasion)

2. Coverage on the activation of OpenAI advertising in European markets

3. Ofcom and requests for access to encrypted messaging on Telegram (contextual reference on the gradual erosion of security)