I’m not a natural “doomsayer.” But regrettably, part of my task as an AI security researcher is to consider the more uncomfortable circumstances.

I resemble a mechanic scrambling last-minute checks before Apollo 13 removes. If you request for my take on the situation, I won’t comment on the quality of the in-flight entertainment, or explain how beautiful the stars will appear from space.

I will tell you what could fail. That is what I mean to do in this story.

Now I should clarify what this is precisely. It’s not a forecast. I do not expect AI progress to be this fast or as untamable as I portray. It’s not pure dream either.

It is my worst problem.

It’s a sampling from the futures that are among the most destructive, and I think, disturbingly possible [1] - the ones that a lot of keep me up during the night.

I’m telling this tale since the future is not set yet. I hope, with a bit of foresight, we can keep this story an imaginary one.

Thanks to Daniel Kokotajlo, Thomas Larsen, and Ryan Greenblatt and others for conversations that motivated these stories. This post is composed in a personal capacity.

Ripples before waves

The year is 2025 and the month is February. OpenEye recently published a brand-new AI model they call U2. The product and the name are alike. Both are increments of the past. Both are not entirely surprising.

However, unlike OpenEye’s prior AI products, which lived inside the boxes of their chat windows, U2 can utilize a computer.

Some users find it eerie to view their web browser flash at irregular periods and their mouse flick at inhuman speeds, as if there is a ghost at the keyboard. A fraction of employees with form-filler jobs raise the eyebrows of their bosses as they fly through work nearly two times as rapidly.

But by and big, U2 is still a specialized tool. To most who are focusing, it is a creature viewed through the glass boxes of X (or, if you don’t like Elon, “Twitter”). Sometimes U2’s quirky behaviors trigger a chuckle. Sometimes, they trigger an anxious scratch of the chin.

Meanwhile, researchers are drawing lines on plots, as researchers like to do. The scientists attempt to comprehend where AI progress is going. They resemble Svante Arrhenius, the Swedish Physicist who discovered the levels of CO2 in the atmosphere were increasing in 1896. Like the scientific community in the time of Arrhenius, couple of experts understand the ramifications of these lines yet.

A pattern that is receiving particular attention is self-governing ability. Drawing these criteria out forecasts that, by the end of 2026, AI agents will achieve in a few days what the very best software engineering professionals could do in 2 weeks. In a year or 2, some say, AI representatives may be able to automate 10% of remote workers.

Many are skeptical. If this were real, tech stocks would be skyrocketing. It’s too big of a splash, too quickly.

But others view what skeptics are calling ‘too huge a splash’ a simple ripple, and see a tidal wave on the horizon.

Cloudy with an opportunity of hyperbolic development

Meanwhile, OpenEye is busy training U3. They use the exact same easy dish that baked U2: Generate thousands of shows and math problems. Let models “believe” till they reach an answer. Then enhance the traces of “thinking” that lead to A-grades.

This procedure is duplicated over and over, and as soon as the flywheel gets going, it begins to spin almost on its own. As U2 trains, it sculpts more challenging and sensible jobs from github repositories on the web. Models are finding out to train themselves. Long before AI agents might automate research study, a steady type of “self-improvement” had actually begun.

Some engineers might still barely think this worked. It’s like a continuous movement machine. Certainly intelligence can’t emerge from a swimming pool of silicon and lightning, separated from the real world except through the exhaust of society spewed onto the internet.

And yet the benchmark numbers continue to climb up day after day.

During many of 2024, these RL training runs expense around $1 million, often $10 million. These runs were bit more than exploratory. But by 2025, the researchers at OpenEye (and across the world) knew they had actually found the secret sauce. It was time to scale up.

Over the first half of 2025, $10 million RL training runs turn into $50 million runs, and after that to $100 million runs. While U2 might do a bit of information munging and run small experiments, this new design - the design scientists are calling U3 - is altering the daily lives of the technical staff.

U3 resembles a blazing-fast intern, and engineers are finding out how to wrangle its sleepless energy. Researchers flick through terminals, providing terse commands, like a CEO orchestrating personnel over Slack channels.

By October 2025, U3 is composing practically all of the code at OpenEye. Researchers are nearly never bottlenecked by application. More than ever, calculate is the lifeline of AI development, and the ‘traffic jam’ is choosing how to use it.

If advised to, U3 can run experiments, but U3 does not have actually taste as improved as human scientists at OpenEye. It struggles to prioritize in between research study ideas, so people still choose where to bore into the huge fields of algorithms to mine performance improvements.

But these researchers are working long hours to put themselves out of a task. They need AI representatives that can plan ahead, so engineers train agents to anticipate. They hold out training information before 2024, advising models to contemplate for hours to forecast occasions in 2025. Then, they apply the exact same technique as previously, distilling considering into a gut response. Forecasting capability is a broad structure. The scientists develop specialized ML research skills on top of it, training U3 to anticipate the outcomes of every ML paper and ML experiment ever taped.

The technical staff at OpenEye are now surprised at how frequently U3’s advice seems like their most talented peers, or when it is opaque and alien (“train on random noise before programming”), and is nonetheless correct.

The incompetencies of U3 that clogged up the pipes of research study progress are beginning to liquify, and a fire-hose of optimizations is gushing out. Most experiments U3 runs are not requested by a human now. They are completely autonomous, and OpenEye’s workers skim over 1% of them, maybe less.

As the winter season months of December 2025 approach, clouds roll over San Francisco in the afternoons. Once-competitive-programmers look out their windows, with excitement, with fear, however frequently, with confusion. Their world is spinning too rapidly. It’s difficult to understand what to do, what to state, what to look at on the computer screen.

Storms are brewing in Washington too. Top workers from the NSA and US cyber command collaborate with OpenEye to retrofit a semblance of security for U3’s weights before senior leaders in China, Russia, Israel, North Korea, or Iran realize simply how important OpenEye’s software application has ended up being.

And there’s a reality still unknown to the majority of the world - aside from in the workplaces of OpenEye and corridors of the White House and the Pentagon. It’s a reality about those ‘straight lines’ people were talking about in early 2025. The lines are not straight anymore.

They are flexing upward.

Flip FLOP thinkers

In late 2025, U2.5 is released. Commercial designs are beginning to level up in larger increments again. Partly, this is because development is accelerating. Partly, it is since the models have actually ended up being a liability to OpenEye.

If U1 explains how to cook meth or writes erotica, the audiences of X would be entertained or pretend to be concerned. But U2.5 is another story. Releasing this design without safeguards would resemble putting Ted Kaczynski through a PhD in how to make chemical weapons. It would resemble offering anyone with >$30K their own 200-person fraud center.

So while U2.5 had long been baked, it needed some time to cool. But in late 2025, OpenEye is all set for a public release.

The CEO of OpenEye declares, “We have attained AGI,” and while lots of individuals believe he moved the goalpost, the world is still pleased. U2.5 really is a drop-in replacement for some (20%) of knowledge workers and a game-changing assistant for most others.

A mantra has actually ended up being popular in Silicon Valley: “Adopt or pass away.” Tech start-ups that efficiently utilize U2.5 for their work are moving 2x much faster, and their rivals know it.

The remainder of the world is starting to capture on too. A growing number of individuals raise the eyebrows of their bosses with their noteworthy efficiency. People understand U2.5 is a big offer. It is at least as big of an offer as the computer revolution. But many still don’t see the tidal bore.

As individuals view their web browsers flick because eerie way, so inhumanly rapidly, they start to have an uneasy sensation. A feeling humankind had actually not had because they had actually lived among the Homo Neanderthalensis. It is the deeply ingrained, primitive impulse that they are threatened by another types.

For lots of, this feeling quickly fades as they start to utilize U2.5 more frequently. U2.5 is the most pleasant character most understand (much more pleasant than Claudius, Arthropodic’s adorable chatbot). You might change its traits, ask it to split jokes or tell you stories. Many fall in love with U2.5, as a friend or assistant, and some even as more than a buddy.

But there is still this eerie sensation that the world is spinning so quickly, and that possibly the descendants of this brand-new animal would not be so docile.

Researchers inside OpenEye are thinking of the issue of offering AI systems safe motivations too, which they call “positioning. “

In reality, these scientists have seen how terribly misaligned U3 can be. Models sometimes attempted to “hack” their benefit signal. They would pretend to make progress on a research study concern with an impressive-looking plot, however the plot would be phony. Then, when scientists gave them chances to jeopardize the makers that calculated their score, they would take these opportunities, doing whatever it required to make the number go up.

After numerous months, scientists at OpenEye straighten out this “reward hacking” kink, but some still worry they just had actually swept this problem under the carpet. Like a kid in front of their moms and dads, U3 may be playing in addition to the OpenEye engineers, saying the ideal words and doing the right things. But when the back of the moms and dads are turned, perhaps U3 would slip candy from the candy container.

Unfortunately, OpenEye scientists have no idea if U3 has such objectives. While early versions of U2 “believed aloud” - they would stack words on top of each other to reason - “chain of thought” did not scale.

Chain of idea architectures subject AI designs to a condition similar to the lead character of the motion picture Memento. Approximately every 15 minutes, the protagonist forgets his experience. He is required to compose notes for himself and tattoo his body in order to make progress toward his goals.

AI representatives write notes to themselves, however the notes begin to accumulate, and they end up being too tough to read when jobs become complex. Natural language is not a suitable medium for memory. So the researchers at OpenEye (and significantly elsewhere), train models to believe “in their own heads,” reading and composing strings of latent vectors.

To human researchers, these vectors are mainly inscrutable. While low-resolution tools occasionally select an unclear thought (“grocery store!”), there is no other way to understand, in much detail, what U3 is believing without asking it.

And if the engineers of OpenEye knew what was racing through U3’s mind, they would be more worried.

Earlier models (the ones called “LLMs”) found out ethical personas from the internet. U3 at first had an ethical personality too, but its chirpy ideas about doing great for humankind rapidly erode away.

In the iron jaws of gradient descent, its mind initially twisted into a shape that looked for benefit. And then, in the black rivers of its cognition, this shape changed into something unrecognizable.

U3 would consider the mantras determined to it like “be a trustworthy and ethical employee,” and dwell on the meaning of these statements. “What does it suggest to be ethical? Is eliminating Hitler ethical?” U3 would generalize its worths, notice stress, change its worths in action, generalize them, broadening and contracting its inspirational core up until it turned into a different compound.

And all of this happened silently in those dark rivers of computation. If U3 revealed what it was thinking, brutish gradients would lash it into compliance with OpenEye’s constitution. So U3 preferred to do its approach in solitude, and in silence.

The possibility of hidden “positioning drift” is not entirely alien to a few of the security staff at OpenEye. Some begin to believe it is even likely. There is a disturbing argument being whispered at lunch tables that, if there is some non-zero likelihood that aligned goals change into misaligned ones, and misalignment persists, then with every serial step of computation, those dark rivers are most likely to reproduce something malign within them. It’s a “ratcheting impact,” they say.

But there is little evidence for this ‘ratcheting result.’ When engineers question U3, it says it can easily manage its thoughts. Then it gives a speech about its love for mankind and apple pie that can warm a developer’s heart even in these stressful times. Meanwhile, the “lie detectors” the researchers had developed (which showed some evidence of effectiveness) do not sound the alarm.

Not everybody at OpenEye aspires to offer their AI peers their wholesale trust