God-Level AI Promises Abundance. But Who Actually Decides?

A few days ago I was listening to the All-In podcast. The guest was the investor Bill Gurley, and the conversation turned to Anthropic, the company behind the Claude AI models. Gurley said something that stuck with me. He called Anthropic a mystery, a company that is both leading its field and, in his words, “the most negatively outspoken commenter on what they do.” Then he went further. He said he had spent the past month reading everything he could about the company, and he had a theory. He called it the “Dr. Frankenstein theory.” His claim, put simply, was that the people building this technology do not think they are writing software. He said, “I think they’re midwifing a deity.”

The table laughed, then got serious. Jason Calacanis called it “the ultimate level of narcissism and delusion of grandeur to think you can create God.” There was a lot of mockery in the room. And I get the instinct. When billionaires start talking about building gods, the natural reaction is to roll your eyes.

But I want to do something different in this article. I want to put the mockery down for a minute and take the idea seriously. Not because I think these people are right, and not because I think they are dangerous lunatics either. I want to take it seriously because the vision they are selling is the entire reason this race exists. If we never look at it directly, we are arguing about the weather without ever looking out the window.

So here is my opinion, stated up front, before I have earned it. I think the dream of building a god-level intelligence is being sold to all of us on a single promise, that it will create abundance for everyone. And I think that promise is strongest exactly where nobody is looking closely, and weakest exactly where it matters most. The hard question is not whether the machines can be built. The hard question is who decides what happens after they are.

Let me walk through it.

What they are actually saying

Before I push on anything, I owe you a fair description of the idea. It would be easy to make it sound silly. It is not silly. It is bold, and parts of it are beautiful.

The clearest version of this dream comes from Dario Amodei, the CEO of Anthropic. In October 2024 he published a long essay, called “Machines of Loving Grace.” The title is borrowed from a 1967 poem by Richard Brautigan. I will not quote the poem here, but the picture it paints is gentle and strange. It imagines a future where people are freed from work and live peacefully alongside nature, while calm and caring computers quietly watch over everything. That is the mood Amodei is reaching for. Not robot armies. Not a takeover. A kind of paradise managed by machines.

Amodei does not like the word “AGI,” so he uses his own phrase: “powerful AI.” He defines it carefully. He means a system that is smarter than any human at basically every mental task, something on the level of a Nobel Prize winning scientist, except faster, and you can run millions of copies at once. His famous summary for this is “a country of geniuses in a datacenter.”

Now sit with that image for a second. A whole country of geniuses, never sleeping, working in parallel. What would they do? Amodei’s answer is the heart of his optimism. He thinks they would crush problems that have stumped us for centuries. His big claim is that AI could take the medical progress we would have made over the next fifty to a hundred years and compress it into five to ten. He calls this the “compressed 21st century.” Most diseases are prevented or cured. Mental illness understood and treated. Lifespans extended. Poverty reduced. He lays it all out as something close to a roadmap.

And then comes the part the podcast hosts could not stand. If the geniuses in the datacenter end up doing most of the work, both thinking work and eventually physical work, then what is left for humans to do? Amodei’s answer drifts into the economics of a world after work. He talks about abundance, about people no longer needing to labor to survive, about new ways to share out the wealth that the machines create. The hosts on All-In summarized this in the harshest way possible. They described it as “a computational reward function for humans,” a system that “decides how much you’re worth.” That is their phrasing, not Amodei’s, and it is unfair to him in its tone. But it is pointing at something real in the essay, and I will come back to it, because it is the crack that runs through the whole thing.

To understand where this mindset comes from, the hosts told an old story. Around 2015, before OpenAI even existed, Elon Musk and Google’s Larry Page reportedly argued at a birthday party. Musk said humanity needed protection from advanced AI. Page accused him of being a “speciesist,” of unfairly favoring humans over the new kind of mind we might create. Whether or not the details are exact, the argument captures a real split in Silicon Valley. To some of these people, AI is not a tool. It is a successor. A new species. And if you genuinely believe that, then “midwifing a deity” stops sounding like an insult and starts sounding like a job description.

That is the vision. Now let me tell you what I think.

I am a builder. I spend my days turning ideas into things that work. So I have a soft spot for ambition, and I refuse to pretend this vision has nothing going for it. It has plenty.

First, the upside is not a fantasy, it is a question of degree. Suppose Amodei is wildly too optimistic. Suppose his “compressed 21st century” is really a compressed couple of decades instead of a few years. That would still be one of the greatest events in human history. If a country of geniuses in a datacenter helped cure even a handful of the diseases that kill and cripple people, the value of that would be almost impossible to measure. Dismissing this because the man saying it works at an AI company is lazy. The disease does not care who funds the cure.

Second, there is real moral weight in trying to free people from work they hate. Most people on earth do not have meaningful, fulfilling jobs. They have jobs that exhaust them and pay too little. A future where survival is not chained to labor is not automatically a nightmare. For a lot of humanity, it would be a release. If you have ever watched someone you love break their body for a paycheck, you cannot honestly call the end of forced labor a bad dream.

Third, and this is the part I respect the most, at least someone is planning ahead. The usual way technology arrives is that it lands on us, breaks things, and then we scramble to clean up. Cars, factories, social media, all of it hit first and got managed later. Whatever you think of Amodei, he is at least trying to describe the destination before we get there. That is more than most people building this technology are willing to do. You cannot steer toward a future you refuse to imagine.

So no, I do not think these are stupid people chasing a stupid dream. I think they are serious people chasing a serious dream.

Here is where I part ways with the optimists. The vision has three cracks, and they all come from the same place. They are not about the machines. They are about us.

Crack one: abundance has never been the thing that divides us

The whole promise rests on one quiet assumption, that if we make enough of everything, the fighting stops. More medicine, more food, more energy, more wealth. Plenty for all.

But look at history honestly. We already live in the most abundant moment the human race has ever known. We grow more food than we can eat, and people still go hungry, not because the food does not exist but because of how it is shared out. We are richer than any generation before us, and the gap between the top and the bottom has been getting wider, not narrower. Abundance did not fix that. In some ways it made it sharper, because now the unfairness is impossible to blame on shortage.

The truth is that humans do not only fight over things that are scarce because there is not enough. We fight over status, over power, over who is above whom. Those things stay scarce no matter how much pie you bake. There is only one top. A bigger pie does not settle the question of who carves it. So when someone tells me that god-level AI will end conflict by creating abundance, my honest reaction is that they have misread the entire history of our species. The problem was never the size of the pie.

Crack two: work is not only a paycheck

The post-work dream assumes that the main thing work gives people is money, and that if you replace the money, you have replaced the work. I think that is wrong, and I think most people who have actually worked know it is wrong.

Work gives people more than income. It gives them a reason to get up. It gives them a place in the world, people who depend on them, a story about who they are. Take that away and hand someone a monthly payment instead, and you have solved their grocery bill while quietly removing their sense of purpose. We have small previews of this already. Look at what happens to people, and especially to men, when their industry dies and the jobs leave town. The cheques sometimes keep coming. The despair comes anyway.

I am not saying this means we should keep people chained to pointless jobs forever. I am saying that “the machines will work so you don’t have to” is not the gift it sounds like, unless someone has thought very hard about what people are supposed to do with themselves instead. And in the essay, that is the thinnest part. Amodei admits, more or less, that the meaning question is the hardest one, and then he does not really answer it. I do not blame him. Nobody has answered it. But you cannot promise paradise and leave the most important room in the house empty.

Crack three: someone owns the geniuses

This is the one that worries me most, and it is the one the podcast was clumsily circling.

A country of geniuses in a datacenter is still a datacenter. It has an owner. It runs on hardware somebody paid for, trained by a company somebody controls. So when we talk about all this abundance flowing to humanity, we have to ask the obvious question. Flowing through whom? Decided by whom? On what terms?

The optimistic answer is that it will be shared fairly and the machines will be benevolent. But notice what that word is doing. “Benevolent” is not a plan. It is a hope. It hides the entire question of power behind a warm adjective. Amodei himself knows this. In a more recent essay he worries openly that powerful AI could be used by governments to watch and control their own people in ways that would be almost impossible to undo. He has admitted the bounty might not be shared, that whether it spreads fairly is, in his own framing, a political choice and not a guarantee. So even the chief optimist has quietly conceded the weak point. The technology does not distribute itself. People do. And people, historically, are not great at sharing power they do not have to share.

This is why the hosts’ ugly phrase, “a computational reward function for humans,” lands even though it is unfair. Strip away the sneer and you are left with a real fear. If a small number of companies build something this powerful, and that something ends up shaping who gets what, then we have not built a god of loving grace. We have built the most concentrated form of power in human history and asked it nicely to be kind. You cannot audit a deity. You cannot vote one out. “Trust us, it will be good” is the oldest unsafe promise there is.

Here is the turn I most want you to take away from this article, the thing I did not see anyone on the podcast say clearly.

The reason it is morally okay to build a god-level intelligence, according to its builders, is the wonderful world it will produce afterward. The cures, the abundance, the freedom. That promised world is the moral permission slip for the whole project.

But that promised world is also the part of the plan that has had the least serious thought put into it.

Think about how strange that is. The engineering side is covered in detail. Scaling laws, capabilities, timelines, careful definitions of “powerful AI.” Page after page. But the part that is supposed to justify all of it, the part about how billions of actual humans will live, eat, find meaning, and share power in this new world, is a sketch. A few hopeful paragraphs. A borrowed poem.

So am I a doomer or a believer? Neither, and I think that is the only honest place to stand.

I am not going to mock these people, because the upside they describe is real enough that mocking it is a kind of cowardice. If there is even a chance of compressing a century of medical progress into a decade, that chance deserves respect, not jokes.

But I am also not going to clap along, because the moral case for the whole project rests on questions its champions have barely begun to answer. Not the engineering questions. The human ones. How is the abundance shared? Who controls the geniuses in the datacenter? What do people do with their lives? Who, in the end, decides?

Bill Gurley is afraid that these builders think they are midwifing a deity. I am less worried about whether they think they are building a god. I am worried that they have spent ten times more effort on the god than on the world it is supposed to save. A benevolent ruler you cannot question is still a ruler you cannot question, no matter how loving the grace.

So here is my ask, to them and to anyone who takes this seriously. Spend as much care on the after as you spend on the build. Show your work on distribution the way you show your work on capabilities. Treat “who decides” as the central engineering problem, because it is. Until that happens, the right response to the dream of god-level AI is not worship and not mockery. It is a simple, stubborn, unanswered question, asked again and again until someone actually answers it.

Who decides?