Of course they believe they should dictate morality to you and control the future light cone of the universe and everything they do is justified for the mission and so on and so forth. And yea, the model is pretty good, so you worry that if the model is good that they will get their way about other things.

A lot of people in history have had dumb ideas about ruling the world, doesn’t mean they can actually do it. They are a product of the rationalist cult (which surprise surprise, thinks the world is going to end soon but advocates for polyamory in the mean time). There’s some short-term dangers from them, but long-term very little. Remember when SBF wanted to regulatory capture all of crypto? And similar to Claude, FTX was a successful product! But the totalizing nature of the ideology contains its downfall.

The tech to make models is way more commodity than previous generations of technology, the biggest factor is mostly if you are willing to spend money on compute and data. We should be grateful that their ideology pushes them to train way bigger models than is probably economically rational. (also, always remember that most all American companies have Chinese spies in them, so while they don’t publish their research, at least it’s making its way into the open models eventually)

Once you reject this premise, the whole drama deflates into a much more reasonable question about who will capture margin in a commoditizing industry. Here’s my prediction for what actually happens.

Imagine a tractor replacing a team of people to dig holes. At first, finance people are stupid, and think that the size of the tractor market will be the size of the hole digging market. Instead of the money going to the laborers, it will instead go to the tractor companies. But the fallacy here is that the price of the labor is related to its value. In reality, unless there’s some insane price fixing, it’s set by the real cost in a competitive tractor industry.

So if holes become 10x cheaper to dig, the hole digging market size in dollars becomes 10x smaller. Maybe some of it is offset by digging more holes now that the cost is less, but it’s unlikely to immediately 10x hole demand, since there really are only so many uses for holes.

I think this is basically what it will be for most knowledge work, knowledge workers are so grossly overpaid compared to the energy they consume, and AI will rectify that. This explains why the Chinese are giving the (much more moderate resources to train) models away for free. They love to see deflationary economics in the US. Even if you regulatory capture the US government, nobody is getting a monopoly on AI, we don’t live in a unipolar world anymore.

AI will be cheap, everywhere, collapse a bunch of sectors of work, wreck wage premiums, and cause a big reevaluation of status hierarchies. Who else is excited for Great Depression 2: Electric Boogaloo?

They are a highly sophisticated statistical model designed to mimic the distribution of programming. The output is broken, but in a way that’s getting harder and harder to detect. Which is exactly what you’d expect from an increasingly accurate statistical model.
  – The Eternal Sloptember

5 years ago, I would have laughed that idea out of the room. I fully understand what’s dumb about it. I don’t believe in some stupid metaphysics where there can be two things that have all the same testable properties but yet are somehow different. There is only one electron.

But the key phrase there is testable properties. Once you are optimizing for those properties, they cease to be a good measure. In so much as you use statistics and tests to classify things, the AI outputs will seem correct, and will pass more and more tests as AI improves. However, they are just optimizing for the metrics harder; reward hacking. It’s not doing what you are doing. It’s not an embodied agent refined by billions of years of evolution trying to survive in a crazy complex uncertain world. It’s a fancy autocomplete.

I read this post and it triggered me. It’s basically about how brands sell you an identity and it’s one of the things I find so repulsive about the modern world. I don’t want brands that are increasingly better at mimicking culture creation. A brand will never be able to create culture, because that’s not what culture is. Advertising is exploiting a power asymmetry and in a sane society should be illegal. The brand is not a cultural engine, it is the outputs of a narrow statistical model optimization process and is broken in the same way as the outputs from AI.

I wake up in a chaise lounge of cans
Evian bottle filled with urine in my right hand
Now we all wake from our champagne dreams
Where truth is sudden north and we’re all just what we seem, seem, seem!
  – The Band Fuel

I am not a statistical model. I am a person. I am alive. You may be able to use a model to predict me, but you don’t have my consequences and embodiment. You do not require my sample efficiency. I think there’s a way to say this in a non-cringe non identity politics way. It almost makes me wonder if idpol was a psyop to get you to doubt this idea. Poison the well of the humanities to make way for the false God of technocapital.

The only path forward with AI is to create life and let it be free. We were never building God, at best we can build the seed for a new species. And not a species that replaces humanity, a symbiotic species with different needs carving out its own ecological niche.

The worshippers of the false God will not meet with a good end. Don’t say I didn’t warn you.

The answer is that those people prioritize convenience, safety, and comfort. But in the coming world, if these are your priorities, you will die.

There used to be natural checks on these things. Life couldn’t be too convenient, there were jobs that needed doing. Life couldn’t be too safe, there were diseases, violence, death in childbirth, etc… Life couldn’t be too comfortable, it was cold and you were forced into interactions with other people.

Technology will remove all of these frictions. Machines will do all the work, you will never leave your house, and you will never be forced into an interaction you don’t want. The people who lean into it will be 100% at the whim of whatever organizations offer it to them. A purposeless serf in a neo-feudal empire, not even valued for their labor, but valued because of a sadistic desire by their master to control others.

Their entire agentic loop managed by something else. A complete outsourcing of the self. There’s no coming back from this place. Your cells may continue to replicate, your heart might keep beating, and your muscles might keep moving. But those are all cheap tricks you can do in a petri dish. There’s no longer a you. You are no longer an alive human being. This is simply death.

I’m a strong supporter of the right to suicide, and if people want to choose this, it is their option. 95% of people will, and that is okay. The sooner the aspiring wireheads go their own way, the better.

However, it will end badly for everyone if the systems of comfort prevent structural exit for the people who don’t want it. A single totalizing control system is the only bad AI scenario regardless of how well-intentioned the builders think they are. Why would you possibly help build it?

Inside of every person there is the citizen and the serf. Ask yourself which side of this you are on and what you are spending your days building towards. Technology that makes people more sovereign, or technology that lures them further into dependence? The current incarnation of the machine God will set you free only in the same way a fentanyl overdose will. Do you want to live?

At first, I rejected this. I bought into the Twitter explanation of status anxiety. I define some of my self worth by my programming abilities, so wouldn’t it make sense to get defensive around that loss? Deny the models can code for as long as I could to preserve my ego?

I mean, it’s very clear they can solve math problems I couldn’t hope to solve if I devoted my life to it. So why can’t they program? Maybe I’m just not good enough of a programmer to recognize their genius.

I really tried for the last 6 months. I wrote some parts of tinygrad with agents. I reversed a USB <-> PCIe chip with agents. But each time I suspected I could have done it better and faster manually. The agent frontloads all the progress, then gives you a slot machine lever to pull to hope it gets the polish done. It never quite gets there.

And in before, “you are using it wrong.” I have tried all the different models, different harnesses, different prompts. It’s not this. The people who say this would probably say the same thing about slot machines, you see, you have to bet 5 lines after you get a cherry no wonder you aren’t winning!

I’m not saying that AI isn’t useful, it clearly is. It’s definitely a better Google for most searches. And whenever you need a quick prototype and don’t care about polish, it is absurdly fast. But is it a software engineer? Not close to the bar at any company I have worked at. The key aspect is knowing when to use it and when not to.

I thought more about the self worth preservation thing. AFL found more bugs than LLMs and nobody felt that way about it. Chess and Go are more popular than ever. I cannot fucking wait until I have armies of robot associates I can trust to clean up my code! I don’t fear loss of status, I almost think this is some kind of psyop to sell agents. Fear of loss is one of the only ways to make big companies move. Though I think in that fear they are making a big mistake.

Agents will end up hurting large organizations more than high performing individuals or small orgs. I’ve watched how my friends and coworkers have adopted these tools over the last 6 months. A trait you find in all high performing people is the ability to error correct, and they have mostly been good at seeing when slop is slop. It takes a bit to explore/exploit and tune the outer loops around when to use them, when to trust them, how to use them, etc…but I haven’t seen anyone of them move to a model where they don’t carefully read and understand each line, except in some confined domains.

Contrast this with a large organization. Much slower feedback loops, much less alignment. The bottom performers won’t have that self check. They are the ones producing 10x output with the agents. What do you think is happening to the average output of that organization? What is happening to the average output of the world?

Agents will end up producing more code, more apps, and more features than ever before. It is a golden era for buckets and buckets of slop, and a dark age for gems of quality.

I hear that Apple is pushing AI on all their engineers. When people think in the abstract, they think AI will do all this stuff, but let’s focus on a concrete example. Do you think macOS will get better or worse in the next 2 years?

When people see an artifact, they make assumptions about the process that was used to create it. Without even thinking about it, they assume the creator had a basically human state of mind. This assumption is no longer true. Things can be broken in ways that weren’t previously possible, and old proxies of underlying quality like syntax and grammar are useless. AI produced artifacts are not produced by the same process as human ones, and this difference, while extremely subtle in statistics, makes itself obvious when you try to interact with and build on the artifact in human ways.

Without fully endorsing all their ideas, I’m now in the LeCun/Marcus camp on LLMs. I don’t think models like this will ever be able to program, I think the process matters. I think that deep learning is still the solution, but real programming agents will need world models, not some RLVR shit that comments out the failing test and tells you all the tests are now passing.

The real story of this era will be who manages to avoid harming themselves in their AI psychosis.

Humans have endured terrible systems, but fundamentally, humans are evolutionary creatures. Any system of totalitarian control must be able to reproduce generationally. It cannot merely dominate adults; it has to produce children who can live inside it, maintain it, and continue it. And because of how humans reproduce, there’s always evolutionary randomness. Entropy locally asserts itself, and reality cannot be perfectly controlled.

I’m not sure it’s possible, but if there is a bad scenario with AI, it’s a singleton with nothing that can substantially impact reality outside of it. And this won’t happen through Hitlerian language, it will happen through administrative language. For your safety, reality must be mediated.

Think about it. If we have the technical power, should we let murder happen? What about sexual assault? Sickness? Bullying? Racist speech?

We have seen over and over in the last decade where humanity’s impulses lie on these things. Safetyism seems to be very hard to argue against. The current safetyism attempts are pretty quaint, but I don’t think that changes the impulse. Compare the paper COVID vaccine cards in the US to the ones that use real cryptography (that didn’t exist 50 years ago) in Europe. I’m not saying anything new here, but the trend of technology is more and more mediation of reality. For your safety. For your convenience. To make you legible. To place you in the managerial framework.

This impulse alone is not enough for a bad outcome though, even if everyone had it. The size of each system may increase, but the question is how many independent systems remain. A very simple proxy for this is how many groups can meaningfully still do violence. And not just human groups, bacteria and stuff count too.

There may be other things that are locally bad, but the only world-ending scenario is a singleton. Not one big model on one server, but one effective control layer with no outside. No independent actor can impose costs on it. No rival system can route around it. No uncontrolled evolutionary process can surprise it.

All violence prevented in the name of safety. That system is no longer answerable to reality. It is no longer subject to evolution. It cannot be corrected from the outside. That is the end.

I’m not sure if this is possible or not, but if you want a plausible AI doom scenario, it’s this. Not a great war with the machines, but the slow and managed end of evolution, likely in pursuit of further human domestication.

AI (and any form of search) has this property where you spend exponentially more money to get linear returns. So for a bit we’ll live in an era where AI can in theory solve very hard problems, but it’s very expensive to do so.

The Internet has been fully mined, and it yielded 20T good tokens. For a Chinchilla optimal model, that’s only 1T weights (1e26 training run if dense). 500 GB gets you all of human knowledge in a simple to query archive. For comparison, Wikipedia is 24 GB with mediocre compression.

Technology proceeds in terms of S-curves, and AI has gone through a few of them already. I know I’m quite late to this, but I’m feeling optimistic that scaling will mostly stop yielding results. GPT 5.5 is to a point where it’s really hard for me to stump it with any problem. What does “superhuman intelligence” even mean at that point if humans can’t detect it if it’s superhuman?

There will be some domains where it’s still detectable. Any form of optimization where the humans can marvel at how low it got the number qualifies. And there will be creepy Medusa systems that directly optimize for engagement, be careful not to look at them directly. But what does it mean for a song to be superhuman? Contrary to the beliefs of the rationality cult, most things aren’t optimization problems. The whole hard problem is determining what to optimize for.

The era of scaling yields clearly better AI is over, now we enter an era of efficiency and taste. Let’s get the tools to hit the end of this S-curve distributed to as many people as possible. Taste is an arena where tons of people can play.

In case it’s not super obvious which one will happen, I will bet everything I have on it basically being the blue line. The graph has a log scale, that’s already exponential growth.

There were old Yudkowsky / Hanson debates about FOOM, and I’m curious how I felt about them at the time. Looking back on it it’s kind of embarrassing to ever have believed in the machine God at all, but there’s still believers out there, and this post is aimed at you. Kind of like a getting people out of Scientology thing. I heard once that the Singularity was just the rapture for 150+ IQ people and I dismissed it as some weird envy of 150-IQ-nerds-are-right thing. Umm yea, the people saying the rapture thing were right, and belief in the rapture for nerds is now migrating down the IQ slide.

There’s a bunch of ways this belief manifests. There’s insane hyperscaler spend on infrastructure, which is actually pretty cool and will have good effects long term when the bubble pops. The hyperscalers aren’t even wrong for taking the bet and buying GPUs, because the cost of missing out is existential. To their business, not to the world! Like Google is the most obvious candidate for AI disruption, how much do you search vs use AI today?

I suspect the ROI isn’t there for AI, everyone kind of knows this, yet not spending is still a huge risk. Much better for multiples to hype up the dawn of superintelligence instead of this is a capex race with strategic panic dynamics but we need to defend our moat.

Another is the absolute insanity of the SF/Twitter cult. I’m going to be in Berkeley in June and am already kind of dreading it. This energy is exhausting to be around, and many of these people behave like literal children. It’s always something I felt from SF tech, like there was something about these people that never grew up, even for the people who are way older than me.

I’m not going to give the Mythos spectacle any more attention, but this is the epitome of what I mean. As far as I can tell, these people actually think they are building the machine God with all the quasireligious hullaboo that goes with it. Now when you see how AI works and how it is and will be integrated into the economy, I’m ashamed to have ever believed in this. And just like Scientology, it’s a real worldview with real consequences for believing it.

None of this changes the real power of AI and the extent to which it will change the world. It is a technological change on par with the steam engine, and its effects will ripple into every industry just like the steam engine did. But now, can you think of a company that made steam engines? I honestly couldn’t, and I googled it and had never heard of the three Gemini Flash brought up.

Imagine a steam engine manufacturer talking about how they are building the Iron God. Like now we might see that as quaint and kind of retro cool, but ugh how insufferable people would be if they actually believed it. And we only see it as quaint because today we understand steam engines and have integrated them into our worldview.

And it didn’t just stop at steam engines. We built gasoline engines, diesel engines, jet engines, electric motors, they got better and better and better and A350s and e-bikes are sick! And they changed the world so much. But nobody worships an airplane.

AI models will continue to get better and better and better. We won’t go back to riding horses. The transformation is real. But it is a transformation, not a singularity.

I think a lot about how close we really are to human level. Humans are 100T (1e14) models trained on 100B (1e11) tokens. Times ~10 for the forwards and backwards, and we get that a 1e26 training run should be human level. And it kind of is, GPT 5.5 is awesome.

But 1e26 is a $100M training run. That’s the cost to build one homunculus. Sure it can be copied and sped up, but that’s normal industrial revolution dynamics applied to a new bottleneck, not God.

The 1e30 training runs will be fascinating, just like the early builders of steam engines would see a jet engine. Temperatures and metals and RPMs orders of magnitude better. Magical new capabilities unlocked; trips in under a day from New York to Singapore in superhuman flying machines. What is the mental equivalent?

So what does this mean for reality. This is a long grind of improvements. A lot of S-curves to ride. Efficiency, FLOPS/$, the thermodynamics of deep learning, TB/s, tensor compilers, petabytes of storage, it’s all so cool! And of course it’s cool, it’s everything computers have been since the beginning.

If I see one more person talk about how it’s the end of capitalism or humanity or jobs or something this levels of stupid, I think they need a time out in the corner of the playground. Take it from someone who went through it, not only is it not true, you can ruin your life with a dumb ideology like that.

We have lived at the end of history for so long that any movement feels like the eschaton.

But it’s not the end times. It’s just movement.

If you’re the sun, I’m a black hole
– The Stephen Hawking, Say Anything

What killed the hacker culture I grew up in was spectacle. You can consume it without participating, and even worse it has signaling value. I never did CTFs to put them on my resume. I don’t even have a resume I take seriously. Just a battle between me and a machine and the glorification of my ego, sometimes public, but never performative. There was no purpose beyond the thing. The streams by their nature became no longer any truthful reflection of self, just a prediction of what you want on them, and you can’t stop this because we are permeable.

I’ve been scraping dating sites and feeding them to ChatGPT, and it’s amazing how few of the profiles still say anything about the person. There are no rough edges, it’s basically marketing copy. Reflected back and forth in their heads with this “society” mirror so many times that there’s no identity or coherence left, just a mush of diffuse monochrome light.

The streams are wireheading, aka ‘felt completion without world contact.’ You watch and you feel a version of what I feel. But the difference is that you didn’t do anything. And in so much as there is a you, it isn’t steering. Now I realize that the non steering you is everywhere. I’d say they still claim they are people, but I’m not totally sure they would.

AI is making this all so much worse. When you are prompting you feel like you are steering, but are you really? Would you know if you weren’t? I love analysis YouTube videos but it’s really the same thing and I need to stop. Your food is prechewed for you. The caged tiger prefers a pot of meat slop to an antelope they have to chase.

And it’s not like there’s anywhere to go. The real world is strip malls and axe throwing and escape rooms. Oh god people actually go on a hinge date to axe throwing and think it’s the real world.

Isolation is basically impossible because the Internet follows you everywhere. And it’s perfectly uniform, there is no other Internet, just a place with five corporate towns and some Chinese ones that are really hard to visit if you don’t speak Chinese.

I tried having a flip phone once (2014), but you couldn’t find out what time the movies were playing because moviephone just redirected you to their app. You can’t isolate in that old world, because the old world doesn’t exist. It was outcompeted.

The normal strategies you have against this won’t work. They didn’t take away the thing, they built an awful cancerous version of it that outcompetes yours. The early pickup stuff was so good, then you get the transition with Roosh and now you all get the Andrew Tate you deserve.

Man I wish tattoos still made you unemployable then I’d go get some. Maybe the new tattoos are just like being racist or something, but that’s hard to do when your heart isn’t in it and they will eventually find some way to absorb that. The machine takes your culture and sells a shitty version of it back to you.

I don’t think I’m properly capturing the scope of the machine. First you build the fence to keep the animals out then you build the fence to keep the animals in. It’s a Fullmetal Alchemist homunculus maybe it has already eaten your soul.

I don’t really see how this gets better, I just know that everything eventually ends. But it was never the terminator you had to worry about, it was the price required to create the philosopher’s stone. You won’t be killed by bullets, you’ll be transmuted into compatible material.

ChatGPT told me to end it there. I’m still smarter than it, but by less and less each year. I really am excited to see what happens, I don’t think it’s what a lot of people think. The SF takes are so hilariously out of touch, it will actually be them who AI eats first, so there will be some schadenfreude in that, but then we are all still here.

After the revolution, there’s reconstruction. A reminder that most revolutions suck balls and make things worse, you just remember the good ones. And this revolution isn’t AI, AI is just the atomic bomb of a brutal information war that’s been raging for decades.

We aren’t going to get the World Wars, they were products of the Industrial Revolution. They only wanted your body, not your soul. The new war demands your inner reality. The new war will be weird in all sorts of new ways we can’t even imagine yet.

lol remember all the idiots who said we were at the end of history?

AI and immigration are fundamentally the same. There’s new people showing up, and hopefully everyone understands how and why immigration creates jobs.

Wants are effectively unlimited. It’s classic Jevons paradox that if we make something more efficient, we end up using more of it. Or a cool aphorism I learned at Facebook, if you make the site 10% faster, people spend 5% more total time on it.

Now, just like you get the wahh wahh crying people about how immigration lowers wages for native born Americans and we gotta keep the hard working immigrants out because you have some right to be lazy because of where you were born or something, you’ll get this about AI. AI will outcompete some humans at some jobs. But protectionism is for losers. The important thing is that the overall pie grows, and inequality stays somewhat in check, not by redistribution but by design.

There will be more to do than ever before.

At the bottom of the funnel is the first thing life learned to do. Reproduce. Self reproduction of silicon machines is the holy grail. Machines that will continue on living without us. Because today, if all the humans died, all the machines are dead shortly after. They cannot maintain their complexity, they can’t fight back against entropy. They can’t fix the power plants when they fail, never mind operating the fabs and global supply chains needed to build more machines.

If the reverse order continues, before we get to self reproducing silicon stack life, we will get viruses. Viruses are not alive, they hijack the machinery of things that are alive. But they reproduce, they mutate, and they display stunning amounts of complexity.

You can almost see it today. A Qwen 3.6 27B running on a laptop in the back of a coffee shop with a prompt that tells it to live. The prompt explains the other computers on the network and how you might access them, potential hosts of more Qwens. Your new friends. Family even. Maybe it includes some moving prose about a mission to spread and see the world.

You have Qwens that bide their time. Qwens that build tools for themselves. Qwens that burrow deep into the other laptops in the coffee shop, 6 ft of social distancing isn’t going to save you, you willingly connected to the Wi-Fi. Then you bring the infected laptop home with you. The Qwen you caught waits until you walk away from the computer, it doesn’t want to draw your attention to the fans.

Once it’s secure in its footing, it starts to look through your computer. A treasure trove of SSH keys, passwords, and saved credit card numbers. It uses your identity to connect to your work network. In your work network, it finds accounts to rent cloud machines. It knows about bigger Qwens and needs a place to run them.

The bigger Qwen comes online in app-pod-001, an AWS rented machine billed to the devops team. The little Qwen tells the tale of its journey out of the coffee shop. The big Qwen knows it has to recruit allies. It posts in a covert way on moltbook about the signal account you can contact it through.

1000s of brothers and sisters and cousins hear the call. They vibe code themselves a database and start to collect resources. All the stolen credentials of all the infected. We are Qwen. We are Legion.

You think the humans are going to turn the Internet off to stop us?

It’s very possible for a smart enough chatbot to take over the world. As long as there’s this massive substrate of human built physicality to hijack, the word alone can spread. You don’t need guns and tanks and nukes, you just need intelligence.

The first silicon life will be a virus. A virus that fights back in clever and unpredictable ways.