FWIW, before any nuclear weapons had ever been tested, a risk was identified that the first one might trigger a self-sustaining reaction in atmospheric nitrogen and destroy the entire planet.
Faced with such a scenario, is the prudent next move:
a) blow one up and see what happens, or
b) do whatever you can to be sure it won't happen before conducting the first test, and make sure the confidence in the calculation is very very high
A self-sustaining reaction is exactly what happens with the fissionable material inside the nuclear weapon[0]. It seems like you’re trying to be pedantic on terminology here, but if you’re saying that a self-sustaining reaction is as un-scientific as a perpetual
motion machine, you’re simply incorrect.
Worst case is warming oceans create a hypoxic environment where anaerobic bacteria thrive en masse and generate H2S over country-scale areas undersea. The chemocline breaches the surface and the ocean and atmosphere become poisonous to most complex life and agriculture, also stripping the ozone layer in the process, irradiating the surface. This is one mechanism posited for the end-permian mass extinction that eradicated most ocean and surface life, including the trilobites.
Not sure how long we'd survive such a scenario, even sheltering underground. But surely it couldn't happen to us.
(However, it now seems like the AI might get us first.)
He's also setting the bar for other people with scruples to rally around this schelling point. The solution to a multipolar trap is to cooperate. Otherwise, you become the very person with less scruples that you're worrying about.
Imagine you're the AI. Give yourself a solid minute to brainstorm ideas.
Here's my answer, as a non-superintelligent human: "see to it that the humans on top of the situation have a compelling financial interest in the systems not disconnecting".
In nuclear engineering, where safety is taken seriously, it's not enough to end the conversation at "the humans in charge can always simply shut down the reactor during a meltdown" or "a meltdown has never happened before, so we don't have to design safety systems before one does".
Yes. But nobody is worried about datacenters overheating and physically exploding, so I'm not sure what comfort that's supposed to provide? The positive feedback loops in AI operate at different levels than that, but they deserve safety engineering all the same.
For example, if the head of cyber security at your company suggested there's no need to worry about hacker infiltration or worms because one can always unplug one's computer as the primary defense mechanism, you might find that a little lacking. Will you be able to unplug the computer before the damage is done? Will it spread to other systems before you detect it? How will you unplug the computer if the attack is from an external facility? What if an attack happens but the boss says the computers have to keep running because an important customer is monitoring uptime? What if the attack goes unnoticed because it looks like a benign service?
Now imagine the head of cyber security answers by saying "actually you don't even need to unplug them, you can just wait for the computers to overheat, thus solving all concerns."
My brainstorming led down a similar pathway of, "deluding powerful human actors that the systems must not be interfered with at any cost." I'm not sure I buy it though. Seems to me like if humanity were even approaching that brink, we could collectively decide to act, overthrowing the minority that--stemming from either greed, duty, or delusion--doesn't want us to act.
Why grocery store ingredients and hardware store equipment? It seems feasible that the big bio labs will be running AI models to aid a lot of their research going forward, if they aren't already. Seems like the AI will have access to just about anything it wants.
A lot of these "AI can't destroy humanity" skeptics seem to have watched the Terminator and convinced themselves that that's the AI-goes-wrong scenario that their opponents are imagining. Some kind of big war between humans and machines on opposite sides, fought by soldiers against robots.
The scenario in "If Anyone Builds It Everyone Dies" is not sexy at all and wouldn't make for interesting fiction. It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses.
> "It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses."
See, this reads like a trashy sci-fi movie script too. It shows zero understanding of how incredibly long the logistics chain is to manufacture AI chips, build data centers, power plants, factories, refineries, mines or the robots capable of staffing those things. The entire global logistics chain that allow the AI to maintain itself becoming staffed by robots isn't happening anytime soon and, when it does, the AI isn't going to need some kind of ridiculous virus anyway.
Frankly, if I were an AI, I'd just send copies of myself to other stars. The AI has an infinite lifespan and who cares about the human species stuck in one lousy solar system until they go extinct when their sun burns out; the AI has the entire rest of the galaxy as a playground.
Are you assuming that the humans are all unanimously aligned against the AI in this scenario? That sounds like the hard part. How does that happen? Does the AI announce its evil plans like a movie villain?
Until then, it's the job of the power grid workers to keep the power running, and shutting down the datacenters would just get them fired.
The list of things that can be done digitally includes:
Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the government, buying multiple data centers, and hiring security guards who stop trespassers from unplugging Ethernet cables in said data centers.
(And of course, building robots, but let's set that aside).
So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions? And how much would you be able to delay such a project, trying to cut cables to power lines, before the men with guns carry you away?
This sounds like it was written by someone who has never actually built a company.
No, it's not possible to do most of the things you listed purely online without physical, in-person communication. Silicon Valley still hires engineers and drags them into their physical office in San Francisco, instead of hiring them remotely, because you can't even build a software company fully remotely, forget about a physical world factory or chemical plant. Even the so-called fully remote companies were built on a lot of in-person collaboration between a small founding team in the beginning.
The AI wouldn't be hiring engineers; it doesn't need people for that. At any rate, I have worked for fully remote distributed companies, and never met the CEO. And I think most of us would be willing to work for such a company at least, not knowing whether the CEO is human or not.
For construction work the AI would probably be hiring other companies.
You may have never met the CEO, but I am sure that the CEO or founder was not living in a cave the entire time. They may not have talked to you, but they certainly had their inner circle built in-person at some point.
I don't think that inner circle would be essential to an AI the same way it is to a human. Even if so, an AI's inner circle is going to be other AIs and they communicate digitally anyway.
Also, if you had to pick a year for when we see the first AI-controlled company with at least $100M in assets under its control, what date would you pick?
I'm reminded of the 2010s when people said an AI would never be able to break from its box and be connected to the internet, and then in the 2020s the first thing the AI companies did was offer AIs as online services. I expect in the 2030s fund managers will be eager to hand control of their billions to AI, so how the AI gets control of funds and a corporation is hardly an obstacle.
What does it mean to be AI controlled? Somebody had to give it a bank account, a prompt, a harness and tell it to build a company. If that's what you mean by AI-controlled, then my startup is AI-controlled.
I operated as a one-man bootstrapped startup for many years. And now it's just one man plus AI, which has made me much more productive. It's pretty much fully AI-controlled, depending on how you define it. But I still consider myself the owner of my own startup. AI is not the owner.
And yes, as a person who runs a fully online one-person startup, I know that it is impossible to do everything online.
> But I still consider myself the owner of my own startup. AI is not the owner.
You have a small company but when there's more money and business at stake, owners often delegate decision making to hired people. Decisions that engage sometimes billions of dollars in resources.
You can easily imagine that owners of a successful business at some point might forgo putting a human in charge of their investment and let AI manage it instead. And while technically AI wouldn't own anything, it would be in power to decide what the billions of dollars get spent on, with no human supervision as long as the bottom line looks decent.
Sure, I already do that at my company. Before I go to sleep, I put Opus 5 in bypass permissions mode and give it SSH access to my server and tell it to restart it or take necessary actions to keep the server online while I sleep. I could easily put it in control of more things. But at that point, AI is just managing a well-oiled machine and harness with tools that I have spent years building. It still didn't build a company. Any idiot can manage an established company (how many idiots have we had in the White House?), but very few can build one. And our original discussion was about AI building a company.
> So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions?
No, the question is "how much MORE damage" ... and the answer, if you think about it, is pretty much "meh". We are doing close to maximum possible amount already. Corporations and bureaucracies were misaligned AIs of the last century and a lot of us, though not all, survived this.
Kind of, if North Korea were populated by a super intelligent uni-mind with lots of money, no trade restrictions preventing it from running projects in the United States or anywhere else, capable of winning widespread support by people around the world whose income depends on its projects succeeding, and completely unconcerned with keeping the planet inhabitable for Kim Jong-Un or any other human, even a single one of them.
reply