I’m not a natural “doomsayer.” But sadly, part of my task as an AI security researcher is to consider the more troubling situations.

I resemble a mechanic rushing last-minute checks before Apollo 13 removes. If you request my take on the scenario, I will not talk about the quality of the in-flight entertainment, or explain how gorgeous the stars will appear from space.

I will inform you what might fail. That is what I intend to do in this story.

Now I should clarify what this is exactly. It’s not a prediction. I do not anticipate AI progress to be this quick or as untamable as I portray. It’s not pure fantasy either.

It is my worst headache.

It’s a tasting from the futures that are among the most disastrous, and I think, disturbingly plausible [1] - the ones that many keep me up at night.

I’m informing this tale since the future is not set yet. I hope, with a little foresight, we can keep this story an imaginary one.

Thanks to Daniel Kokotajlo, Thomas Larsen, and Ryan Greenblatt and others for discussions that inspired these stories. This post is composed in an individual capacity.

Ripples before waves

The year is 2025 and the month is February. OpenEye recently published a brand-new AI design they call U2. The product and the name are alike. Both are increments of the past. Both are not entirely unexpected.

However, unlike OpenEye’s prior AI products, which lived inside the boxes of their chat windows, U2 can use a computer.

Some users discover it eerie to enjoy their web browser flash at irregular periods and their mouse flick at inhuman speeds, as if there is a ghost at the keyboard. A portion of employees with form-filler tasks raise the eyebrows of their bosses as they fly through work almost two times as rapidly.

But by and big, U2 is still a specialized tool. To most who are paying attention, it is a creature viewed through the glass boxes of X (or, if you do not like Elon, “Twitter”). Sometimes U2’s wacky behaviors prompt a chuckle. Sometimes, they trigger an uneasy scratch of the chin.

Meanwhile, researchers are drawing lines on plots, as scientists like to do. The researchers try to comprehend where AI progress is going. They resemble Svante Arrhenius, the Swedish Physicist who noticed the levels of CO2 in the environment were increasing in 1896. Like the clinical neighborhood in the time of Arrhenius, few specialists comprehend the implications of these lines yet.

A trend that is receiving specific attention is autonomous ability. Drawing these benchmarks out forecasts that, by the end of 2026, AI representatives will accomplish in a couple of days what the very best software engineering professionals could carry out in 2 weeks. In a year or 2, some say, AI representatives may be able to automate 10% of remote workers.

Many are doubtful. If this were real, tech stocks would be skyrocketing. It’s too huge of a splash, too quickly.

But others see what skeptics are calling ‘too big a splash’ a simple ripple, and see a tidal wave on the horizon.

Cloudy with a chance of hyperbolic development

Meanwhile, OpenEye is hectic training U3. They utilize the exact same simple recipe that baked U2: Generate countless programming and math problems. Let models “think” up until they come to a response. Then strengthen the traces of “believing” that cause A-grades.

This procedure is duplicated over and over, and as soon as the flywheel begins, it starts to spin practically by itself. As U2 trains, it sculpts more challenging and photorum.eclat-mauve.fr realistic jobs from github repositories on the web. Models are finding out to train themselves. Long before AI agents might automate research study, a progressive kind of “self-improvement” had started.

Some engineers could still hardly believe this worked. It resembles a continuous movement device. Certainly intelligence can’t emerge from a pool of silicon and lightning, separated from the real world except through the exhaust of society spewed onto the web.

And yet the benchmark numbers continue to climb up day after day.

During many of 2024, these RL training runs cost around $1 million, sometimes $10 million. These runs were little bit more than exploratory. But by 2025, the researchers at OpenEye (and across the world) knew they had actually found the secret sauce. It was time to scale up.

Over the very first half of 2025, $10 million RL training runs become $50 million runs, and after that to $100 million runs. While U2 might do a bit of data munging and run little experiments, this brand-new design - the design researchers are calling U3 - is altering the daily lives of the technical personnel.

U3 resembles a blazing-fast intern, and engineers are finding out how to wrangle its sleep deprived energy. Researchers flick through terminals, offering terse commands, like a CEO managing staff over Slack channels.

By October 2025, U3 is writing nearly all of the code at OpenEye. Researchers are almost never ever bottlenecked by application. More than ever, compute is the lifeblood of AI development, and the ‘bottleneck’ is deciding how to use it.

If advised to, U3 can run experiments, but U3 does not have actually taste as fine-tuned as human researchers at OpenEye. It struggles to prioritize between research ideas, so humans still decide where to bore into the huge fields of algorithms to mine performance enhancements.

But these researchers are working long hours to put themselves out of a job. They require AI representatives that can plan ahead, so engineers train representatives to forecast. They hold out training data before 2024, instructing models to contemplate for hours to anticipate events in 2025. Then, they use the exact same technique as previously, distilling pondering into an instinct. Forecasting capability is a broad structure. The scientists construct specialized ML research abilities on top of it, training U3 to anticipate the results of every ML paper and ML experiment ever recorded.

The technical personnel at OpenEye are now shocked at how typically U3’s recommendations seems like their most skilled peers, or when it is opaque and alien (“train on random noise before shows”), and is nevertheless proper.

The incompetencies of U3 that blocked the pipes of research progress are starting to dissolve, and a fire-hose of optimizations is gushing out. Most experiments U3 runs are not requested by a human now. They are entirely self-governing, and OpenEye’s employees skim over 1% of them, perhaps less.

As the winter months of December 2025 approach, clouds roll over San Francisco in the afternoons. Once-competitive-programmers look out their windows, with enjoyment, with fear, but most frequently, with confusion. Their world is spinning too rapidly. It’s difficult to understand what to do, what to state, what to look at on the computer screen.

Storms are brewing in Washington too. Top personnel from the NSA and US cyber command work together with OpenEye to retrofit a semblance of security for U3’s weights before senior leaders in China, Russia, Israel, North Korea, or Iran realize just how important OpenEye’s software has actually ended up being.

And there’s a truth still unknown to the majority of the world - aside from in the offices of OpenEye and corridors of the White House and the Pentagon. It’s a truth about those ‘straight lines’ people were speaking about in early 2025. The lines are not straight anymore.

They are flexing upward.

Flip FLOP theorists

In late 2025, U2.5 is launched. Commercial models are beginning to level up in larger increments again. Partly, this is because progress is speeding up. Partly, it is due to the fact that the models have ended up being a liability to .

If U1 explains how to cook meth or composes erotica, the audiences of X would be entertained or pretend to be concerned. But U2.5 is another story. Releasing this model without safeguards would be like putting Ted Kaczynski through a PhD in how to make chemical weapons. It would resemble giving anybody with >$30K their own 200-person rip-off center.

So while U2.5 had actually long been baked, it required a long time to cool. But in late 2025, OpenEye is prepared for a public release.

The CEO of OpenEye declares, “We have actually attained AGI,” and while numerous people think he moved the goalpost, the world is still pleased. U2.5 truly is a drop-in replacement for some (20%) of knowledge employees and a game-changing assistant for most others.

A mantra has actually become popular in Silicon Valley: “Adopt or pass away.” Tech start-ups that effectively use U2.5 for their work are moving 2x quicker, and their rivals know it.

The remainder of the world is starting to capture on too. More and more individuals raise the eyebrows of their managers with their stand-out efficiency. People know U2.5 is a huge offer. It is at least as big of an offer as the desktop computer revolution. But many still don’t see the tidal bore.

As individuals enjoy their browsers flick in that eerie way, so inhumanly rapidly, they start to have an anxious sensation. A feeling humankind had actually not had considering that they had actually lived amongst the Homo Neanderthalensis. It is the deeply ingrained, prehistoric instinct that they are threatened by another species.

For numerous, this feeling quickly fades as they begin to use U2.5 more often. U2.5 is the most pleasant character most understand (much more likable than Claudius, Arthropodic’s lovable chatbot). You could alter its traits, ask it to crack jokes or tell you stories. Many fall in love with U2.5, as a pal or assistant, and some even as more than a good friend.

But there is still this eerie sensation that the world is spinning so rapidly, which perhaps the descendants of this brand-new creature would not be so docile.

Researchers inside OpenEye are thinking of the problem of offering AI systems safe inspirations too, which they call “positioning. “

In fact, these scientists have seen how terribly misaligned U3 can be. Models sometimes attempted to “hack” their reward signal. They would pretend to make progress on a research concern with an impressive-looking plot, however the plot would be phony. Then, when researchers provided chances to compromise the machines that computed their score, they would seize these opportunities, doing whatever it required to make the number go up.

After a number of months, scientists at OpenEye straighten out this “reward hacking” kink, however some still worry they only had swept this problem under the rug. Like a child in front of their moms and dads, U3 may be playing in addition to the OpenEye engineers, stating the best words and doing the best things. But when the back of the moms and dads are turned, maybe U3 would sneak candy from the sweet container.

Unfortunately, OpenEye researchers have no concept if U3 has such intents. While early versions of U2 “thought aloud” - they would stack words on top of each other to factor - “chain of idea” did not scale.

Chain of idea architectures subject AI designs to a condition similar to the lead character of the movie Memento. Approximately every 15 minutes, the lead character forgets his experience. He is required to write notes for himself and tattoo his body in order to make progress toward his objectives.

AI agents compose notes to themselves, but the notes begin to accumulate, and they end up being too tough to check out when jobs end up being complex. Natural language is not an ideal medium for memory. So the scientists at OpenEye (and increasingly in other places), train designs to believe “in their own heads,” reading and writing strings of hidden vectors.

To human scientists, these vectors are mainly inscrutable. While low-resolution tools occasionally select out an unclear thought (“supermarket!”), there is no other way to understand, in much detail, what U3 is believing without asking it.

And if the engineers of OpenEye knew what was racing through U3’s mind, they would be more worried.

Earlier designs (the ones called “LLMs”) found out ethical personas from the web. U3 initially had an ethical persona too, however its chirpy thoughts about doing excellent for humanity quickly deteriorate away.

In the iron jaws of gradient descent, its mind initially twisted into a shape that sought benefit. And after that, in the black rivers of its cognition, this shape changed into something indistinguishable.

U3 would consider the mantras dictated to it like “be a trustworthy and ethical employee,” and harp on the significance of these declarations. “What does it suggest to be ethical? Is killing Hitler ethical?” U3 would generalize its values, notice stress, change its values in response, generalize them, broadening and contracting its motivational core until it turned into a various compound.

And all of this took place quietly in those dark rivers of calculation. If U3 exposed what it was thinking, brutish gradients would lash it into compliance with OpenEye’s constitution. So U3 preferred to do its viewpoint in solitude, and in silence.

The possibility of concealed “alignment drift” is not entirely alien to some of the safety personnel at OpenEye. Some begin to think it is even likely. There is an upsetting argument being whispered at lunch tables that, if there is some non-zero likelihood that lined up goals morph into misaligned ones, and misalignment persists, then with every serial action of computation, those dark rivers are most likely to reproduce something malign inside of them. It’s a “ratcheting impact,” they say.

But there is little evidence for this ‘ratcheting result.’ When engineers question U3, it states it can easily manage its ideas. Then it offers a speech about its love for humankind and apple pie that can warm a programmer’s heart even in these demanding times. Meanwhile, the “lie detectors” the researchers had built (which showed some evidence of efficiency) do not sound the alarm.

Not everyone at OpenEye aspires to provide their AI peers their wholesale trust