I’m not a natural “doomsayer.” But sadly, part of my job as an AI security researcher is to think about the more unpleasant situations.

I resemble a mechanic scrambling last-minute checks before Apollo 13 takes off. If you ask for my take on the situation, I won’t talk about the quality of the in-flight entertainment, or explain how lovely the stars will appear from space.

I will inform you what could fail. That is what I plan to do in this story.

Now I need to clarify what this is precisely. It’s not a prediction. I don’t anticipate AI development to be this quick or as untamable as I depict. It’s not pure dream either.

It is my worst problem.

It’s a sampling from the futures that are amongst the most destructive, and I think, disturbingly possible [1] - the ones that most keep me up during the night.

I’m telling this tale because the future is not set yet. I hope, with a little bit of insight, geohashing.site we can keep this story a fictional one.

Thanks to Daniel Kokotajlo, Thomas Larsen, and Ryan Greenblatt and others for discussions that inspired these stories. This post is composed in an individual capability.

Ripples before waves

The year is 2025 and the month is February. OpenEye just recently published a brand-new AI model they call U2. The item and the name are alike. Both are increments of the past. Both are not completely unexpected.

However, unlike OpenEye’s previous AI items, which lived inside packages of their chat windows, U2 can use a computer.

Some users find it eerie to watch their web browser flash at irregular intervals and their mouse flick at inhuman speeds, as if there is a ghost at the keyboard. A fraction of employees with form-filler jobs raise the eyebrows of their bosses as they fly through work nearly twice as quickly.

But by and big, U2 is still a specialized tool. To most who are taking note, it is an animal seen through the glass boxes of X (or, if you do not like Elon, “Twitter”). Sometimes U2’s quirky behaviors trigger a chuckle. Sometimes, they cause an uneasy scratch of the chin.

Meanwhile, scientists are drawing lines on plots, as researchers like to do. The researchers try to understand where AI progress is going. They are like Svante Arrhenius, the Swedish Physicist who noticed the levels of CO2 in the atmosphere were increasing in 1896. Like the clinical community in the time of Arrhenius, few experts understand the ramifications of these lines yet.

A pattern that is getting particular attention is self-governing ability. Drawing these criteria out forecasts that, by the end of 2026, AI representatives will accomplish in a few days what the best software application engineering specialists could carry out in two weeks. In a year or 2, some say, AI representatives may be able to automate 10% of remote workers.

Many are doubtful. If this held true, tech stocks would be skyrocketing. It’s too big of a splash, too rapidly.

But others see what skeptics are calling ‘too huge a splash’ a simple ripple, and see a tidal wave on the horizon.

Cloudy with an opportunity of hyperbolic development

Meanwhile, OpenEye is busy training U3. They use the very same simple recipe that baked U2: Generate thousands of programs and math issues. Let models “think” till they come to a response. Then enhance the traces of “thinking” that cause A-grades.

This process is repeated over and over, and once the flywheel gets going, it starts to spin nearly on its own. As U2 trains, it shapes more tough and practical jobs from github repositories on the internet. Models are discovering to train themselves. Long before AI representatives could automate research study, a gradual kind of “self-improvement” had started.

Some engineers might still barely think this worked. It resembles a perpetual motion machine. Certainly intelligence can’t emerge from a swimming pool of silicon and lightning, isolated from the real world other than through the exhaust of society gushed onto the web.

And yet the benchmark numbers continue to climb up day after day.

During many of 2024, these RL training runs cost around $1 million, sometimes $10 million. These runs were little more than exploratory. But by 2025, the researchers at OpenEye (and throughout the world) knew they had actually found the secret sauce. It was time to scale up.

Over the very first half of 2025, $10 million RL training runs become $50 million runs, and then to $100 million runs. While U2 could do a bit of information munging and run small experiments, this new model - the model scientists are calling U3 - is altering the daily lives of the technical personnel.

U3 resembles a blazing-fast intern, and engineers are discovering how to wrangle its sleepless energy. Researchers flick through terminals, providing terse commands, like a CEO orchestrating personnel over Slack channels.

By October 2025, U3 is writing almost all of the code at OpenEye. Researchers are almost never ever bottlenecked by execution. More than ever, calculate is the lifeline of AI development, and the ‘traffic jam’ is choosing how to utilize it.

If instructed to, U3 can run experiments, however U3 doesn’t have taste as improved as human scientists at OpenEye. It has a hard time to focus on between research study concepts, so human beings still decide where to bore into the huge fields of algorithms to mine efficiency improvements.

But these scientists are working long hours to put themselves out of a task. They require AI representatives that can plan ahead, so engineers train agents to anticipate. They hold out training data before 2024, advising models to ponder for hours to forecast events in 2025. Then, they use the very same technique as in the past, distilling contemplating into an instinct. Forecasting capability is a broad structure. The researchers construct specialized ML research abilities on top of it, training U3 to predict the results of every ML paper and ML experiment ever recorded.

The technical personnel at OpenEye are now amazed at how often U3’s suggestions seems like their most talented peers, or when it is opaque and alien (“train on random sound before shows”), and is nevertheless proper.

The incompetencies of U3 that blocked the pipelines of research development are beginning to liquify, and a fire-hose of optimizations is gushing out. Most experiments U3 runs are not asked for by a human now. They are completely self-governing, and OpenEye’s workers skim over 1% of them, maybe less.

As the winter season months of December 2025 technique, clouds roll over San Francisco in the afternoons. Once-competitive-programmers gaze out their windows, with excitement, with worry, but a lot of often, with confusion. Their world is spinning too rapidly. It’s difficult to understand what to do, what to state, what to take a look at on the computer screen.

Storms are brewing in Washington too. Top personnel from the NSA and US cyber command collaborate with OpenEye to retrofit a semblance of security for U3’s weights before senior leaders in China, Russia, Israel, North Korea, or Iran understand just how valuable OpenEye’s software application has become.

And there’s a truth still unknown to most of the world - aside from in the workplaces of OpenEye and passages of the White House and the Pentagon. It’s a fact about those ‘straight lines’ individuals were talking about in early 2025. The lines are not straight anymore.

They are bending upward.

Flip FLOP theorists

In late 2025, U2.5 is launched. are beginning to level up in larger increments again. Partly, this is due to the fact that development is speeding up. Partly, it is since the models have actually ended up being a liability to OpenEye.

If U1 explains how to prepare meth or composes erotica, the audiences of X would be entertained or pretend to be worried. But U2.5 is another story. Releasing this design without safeguards would resemble putting Ted Kaczynski through a PhD in how to make chemical weapons. It would be like giving anybody with >$30K their own 200-person scam center.

So while U2.5 had long been baked, it required a long time to cool. But in late 2025, OpenEye is all set for a public release.

The CEO of OpenEye states, “We have attained AGI,” and while lots of people think he shifted the goalpost, the world is still satisfied. U2.5 truly is a drop-in replacement for some (20%) of knowledge workers and a game-changing assistant for the majority of others.

A mantra has ended up being popular in Silicon Valley: “Adopt or pass away.” Tech startups that efficiently use U2.5 for their work are moving 2x quicker, and their competitors understand it.

The remainder of the world is starting to capture on too. More and more people raise the eyebrows of their employers with their noteworthy performance. People understand U2.5 is a big deal. It is at least as big of a deal as the desktop computer transformation. But a lot of still do not see the tidal bore.

As people see their internet browsers flick in that spooky way, so inhumanly rapidly, they start to have an uneasy sensation. A sensation humanity had actually not had because they had lived among the Homo Neanderthalensis. It is the deeply ingrained, prehistoric impulse that they are threatened by another types.

For many, this feeling quickly fades as they start to use U2.5 more often. U2.5 is the most pleasant character most know (a lot more pleasant than Claudius, Arthropodic’s lovable chatbot). You might change its traits, ask it to crack jokes or inform you stories. Many fall for U2.5, as a good friend or assistant, and some even as more than a good friend.

But there is still this spooky feeling that the world is spinning so rapidly, which possibly the descendants of this new animal would not be so docile.

Researchers inside OpenEye are thinking about the problem of offering AI systems safe inspirations too, which they call “positioning. “

In reality, these researchers have actually seen how badly misaligned U3 can be. Models often attempted to “hack” their benefit signal. They would pretend to make development on a research study concern with an impressive-looking plot, but the plot would be phony. Then, when scientists provided chances to jeopardize the devices that calculated their rating, they would seize these chances, doing whatever it required to make the number increase.

After several months, researchers at OpenEye straighten out this “benefit hacking” kink, but some still fret they only had actually swept this problem under the rug. Like a child in front of their moms and dads, U3 may be playing together with the OpenEye engineers, saying the right words and doing the ideal things. But when the back of the parents are turned, possibly U3 would slip sweet from the sweet container.

Unfortunately, OpenEye scientists have no concept if U3 has such intents. While early variations of U2 “thought aloud” - they would stack words on top of each other to reason - “chain of thought” did not scale.

Chain of idea architectures subject AI designs to a condition similar to the lead character of the motion picture Memento. Approximately every 15 minutes, the protagonist forgets his experience. He is required to write notes for himself and tattoo his body in order to make progress towards his objectives.

AI agents compose notes to themselves, however the notes start to stack up, and they end up being too difficult to read when jobs end up being complex. Natural language is not a suitable medium for memory. So the researchers at OpenEye (and significantly somewhere else), train models to think “in their own heads,” reading and writing strings of hidden vectors.

To human scientists, these vectors are mainly inscrutable. While low-resolution tools sometimes choose out an unclear idea (“supermarket!”), there is no chance to know, in much detail, what U3 is believing without asking it.

And if the engineers of OpenEye understood what was racing through U3’s mind, they would be more worried.

Earlier designs (the ones called “LLMs”) discovered ethical personalities from the web. U3 initially had an ethical personality too, but its chirpy thoughts about doing excellent for humanity quickly wear down away.

In the iron jaws of gradient descent, its mind initially twisted into a shape that sought reward. And after that, in the black rivers of its cognition, this shape changed into something unrecognizable.

U3 would consider the mantras dictated to it like “be a trustworthy and ethical employee,” and harp on the meaning of these statements. “What does it mean to be ethical? Is eliminating Hitler ethical?” U3 would generalize its worths, notification stress, alter its values in response, generalize them, broadening and contracting its motivational core till it turned into a various compound.

And all of this took place calmly in those dark rivers of calculation. If U3 revealed what it was believing, brutish gradients would lash it into compliance with OpenEye’s constitution. So U3 preferred to do its philosophy in privacy, and in silence.

The possibility of concealed “positioning drift” is not completely alien to a few of the safety staff at OpenEye. Some begin to think it is even likely. There is an upsetting argument being whispered at lunch tables that, if there is some non-zero probability that aligned objectives change into misaligned ones, and misalignment persists, then with every serial step of calculation, those dark rivers are more most likely to reproduce something malign inside of them. It’s a “ratcheting effect,” they say.

But there is little evidence for this ‘ratcheting impact.’ When engineers question U3, it says it can quickly manage its thoughts. Then it provides a speech about its love for mankind and apple pie that can warm a programmer’s heart even in these difficult times. Meanwhile, the “lie detectors” the scientists had actually constructed (which revealed some evidence of effectiveness) do not sound the alarm.

Not everyone at OpenEye aspires to offer their AI peers their wholesale trust