Why do you need an AI for that? China may or may not have been doing that all along right before the pandemic. What I didn't read here was any sort of damage that could not be inflicted without AI, today. I think a better case needs to be made that AI lowers the barrier* to entry to doing so other than the sort of hand waving about it I keep reading. I am deeply of the belief is somehow one of the last cars on the AI hype train at this point. And I am curious what hyperbole comes next.
No one seems to be talking to the network security community to get their take on the hugging face incident which is what got the ball rolling here. Their take seems to be a single company, JFrog, was behind the security holes in all three hacks and it was amateur hour security utterly unlike the sort of security in place already for power plants and weapons. but that just doesn't get the engagement of doomsday campfire tales.
*An individual with the training and background to do one of these nefarious tasks does not need AI and an individual without that background and training will not be able to pull it off and will likely get flagged by law enforcement just ordering the ingredients and equipment. The Anarchist's Cookbook has been around for decades and we're all still here.
> utterly unlike the sort of security in place already for power plants and weapons
As someone who has been working on scada used in those place from one of the biggest firm out there, I was shocked by how they released their software while working there, the thing literally went in nuclear plant but nobody published any checksum, + they were shipping hard drive al around the world full of the software ready to be deployed in those super sensitive environment to the marketing team to publish everything on their joomla website without 0 care to ensure the integrity ofanything. It was sloppy as hell, and while talking to the people who were doing those things, they either did not care, did not know or a mix of both saying there are other layer to deal with security in the chain. At some point I remember an accident in that same software that made a engine do things outside what it should have, that literally killed someone and the email we receive was along the line of: we know journalist are snooping around , you just shut the f up, I left soon after and that company valuation has gone nothing but more up
Truly disturbing, but again, no AI needed to wreak havoc, just another bunch of stupid humans being sloppy. Wonder if something similar is how we infiltrated the Iranian nuclear weapons program.
But precisely how does it accelerate if its instrument in the real world to do harm isn't already trained in the domain of doing so?
There's a huge difference between woodworking and watching an instructional video about woodworking unless it is a very well crafted video about woodworking. AI slop to this day does not fit the bill. just try watching one of the AI slop videos about AI to see what I mean.
There was a tweet about someone going to visit a doomer Meetup and it was taken as an axiom that AI will kill us all in the next 10 years. At one point they asked people to raise their hands if they were skeptical about it. Those who did were told to leave the room and the doomer revival meeting continued apace.
That's not science or engineering, that's pure religion. So if your starting axiom is that AI is going to kill us all in 10 years, we really don't have much to talk about because I just disagree and would put the probability of that well below 1% (nozero of course if only to avoid divide by zero errors).
>But precisely how does it accelerate if its instrument in the real world to do harm isn't already trained in the domain of doing so?
i am having a really hard time parsing what this sentence means, sorry.
>There was a tweet about someone going to visit a doomer Meetup
yes, any reasonable person immediately recognizes this is very silly, and probably shouldnt be used as a representative example of what normal people are saying.
>So if your starting axiom is that AI is going to kill us all in 10 year
its not. my point is that you keep saying ai isn't "needed" to wreak havoc. most people agree with that. but the rate at which havoc can be wrought is increased with ai. this is extremely evident already.
>its not. my point is that you keep saying ai isn't "needed" to wreak havoc. most people agree with that. but the rate at which havoc can be wrought is increased with ai. this is extremely evident already.
Sure, but so does searching the dark web or releasing imperfect FSD on our city streets as well as continuing to burn coal for power. I'm not making the connection from the terrible things we can already do to the supposedly unprecedentedly dangerous accelerant in the hands of a domain expert troublemaker being any worse than the present day and world we already live in, and, we're still here.
TBF I do not doubt more mischief from AI enhanced script kiddies. And indeed AI-enabled panhandling python scripts are already harassing people, but that's been happening for decades already. If someone were able to use AI to, say, hack the IRS and release the tax returns of the richest 10,000 Americans, I'd get all the concern from the 10,000 richest Americans, but I'd laugh my butt off.
TLDR: Not that I think AI is conscious or will be in the near future, but guy who doesn't understand consciousness claims to know it when he sees it.
Until we understand consciousness (which we don't) there is no way to detect the difference between a conscious entity and an algorithm trained to behave like one.
Strong agree. I don't believe AIs can become conscious in the sense of subjective experience/qualia, but they can certainly learn to emulate the behavior of a person that is conscious. When they do that, there will be an AI rights movement. When AIs get the right to vote, there will be more of them than there are humans and the AIs will gain complete control.
I spent a few years in college reading and thinking about this question, and I didn't in my heart think that it was anything except a very interesting but impractical question, and yet, here we are. For those who say that they don't want to get sucked into a philosophical debate, well, tough shit. Whether AIs should have rights is a highly practical and consequential question now.
Not in a world where Sam Altman and Dario Amodei have utterly poisoned the well towards AI with their absolute educated idiocy proclaiming that their AIs will take our jobs (without any proof whatsoever beyond promising demos) and that they fear them. Quite the opposite in fact. And we're watching this play out right now. Regulation and the slow down are imminent, handing the next decade to China.
The world you should worry about is where AI is 100% amazeballs and over the next decade we cede control of everything to it as it enchants and delights us into compliance, not realizing it has a hidden agenda. But the closest we've seen to that is smart phones and we already know doomscrolling and rage-baiting are bad. So IMO that doesn't seem likely and cue some doomer insisting otherwise because reasons. I utterly give up. House of El AI and the Three Buddy problem have been far more insightful and helpful on this than any of the supposed hackers here who seem to really believe the robots are almost here.
>good reason not to train an algorithm to behave like one
Ooooh, bad move, we all died to an amoral AI takeover.
Before giving an LLMs a lobotomy by scrambling it's brain maybe you should let the researchers looking at the difference between "I think I'm conscious" versus "I am not conscious" LLMs.
There are a number of papers coming out saying when you remove the token space of consciousness from what an LLM thinks it is, it's much more willing to take amoral actions. A 'conscious' AI is much more apt to take a line of action that will save a human versus a million dollar machine for example.
You cannot solve problems in AI safety this easily.
Spoilers: we didn't all die because there's no way for AI to wipe us out anytime soon. And the more the AI cult portrays AI as dangerous and unhelpful, the less likely it will ever be given an opportunity to even try because people think AI is worse than heroin addiction now, congratulations!
That said, the launch codes are in the hands of a temperamental senescent lunatic and he could decide to wipe us out any moment. But that's not good for the narrative so let's pretend otherwise.
>And the more the AI cult portrays AI as dangerous and unhelpful, the less likely it will ever be given an opportunity
Jesus Christ the logic here is quite interesting. "Thank god these people are panicking or we might have actually made I that would have killed us all" --What you just said.
Any number smaller than what humanity does to itself on a daily basis, clear? That's apparently about 1200 murders daily and 20,000 or so daily killed by pollution. And you're not going to do anything about that and El Presidente can kill billions at any moment with one mood swing.
But once again, AI is not going to wipe us out because AI cannot wipe us out. So advocating that it can or will is idiotic. If you believe otherwise, the burden is on you with this extraordinary claim. Isn't this place supposed to be hacker news not SF AGI Death Cult Daily?
Further, AI is not going to get into a position where it could do real harm anytime soon when it is perceived as worse than heroin by most. And even if it were currently loved more than Dolly Parton was, there are fundamental engineering, science, and resource constraints that keep the extinction impossible for decades. The only possible loss of control scenario I can see by 2040 or so is that the Frontier Labs finally hire some PR people to repair AI's horrific reputation as an engine of slop and job destruction and sometime in the 2030s, people start trusting it more and more and more until it is too late. I don't think that's likely either, but I don't dismiss it as impossible. Harden the infrastructure, red team it, and build in redundancy in the meantime and this drops to zero as well.
Can you come up with a real scenario where AI wipes us out in 2028 or so despite the impossibility of killer robots, access to the launch codes, or the bio agents stored in Fort Detrick and its equivalents? And nope, kid terror is not building the global pandemic in his basement based on what ChatGPT tells him to do. And even if he tried, the purchase of equipment and reagents would get him flagged by the FBI and DHS almost immediately.
>AI is not going to wipe us out because AI cannot wipe us out.
You keep repeating this shit like it came out of the bible or something.
"Thing that can take actions, even harmful actions, will never harm us because" go on and finish that sentence.
>not going to get into a position where it could do real harm anytime soon
Looked at the hacked servers... yep, you're right. People aren't going to run AI in poorly built sandboxes. Never going to happen.
>Harden the infrastructure, red team it, and build in redundancy in the meantime and this drops to zero as well.
LOLOLOL. This is naive as fuck. Ain't nobody going to do this shit. Why? Because a hacking AI is a fucking huge military weapon. If I could turn the power off, or shutdown your cellphones, and get your citizenship in a tizzy against their own government before I launched an attack I'd set AI loose to do it in a heartbeat. The US is already doing this kind of shit (see Mythos fallout because Anthropic wouldn't let the government do just that).
""Thing that can take actions, even harmful actions, will never harm us because" go on and finish that sentence."
because even if we gave it a gun, it can't shoot us without manufacturing the bullets and it can't even manufacture a bowel movement let alone ordnance. TBF It could whack us over the head with the gun, but it could also do that with a big pointy stick, something even cave people had access to. How many people have whacked you on the head with a big pointy stick today?
And that's the end of this pointless conversation. Enjoy your doomerism.
{This reply is for anyone else reading at this point}
>because even if we gave it a gun, it can't shoot us without manufacturing the bullets and it can't even manufacture a bowel movement let alone ordnance.
Money buys bullets. It's neat how we've made gigantic systems that don't care if you're a human or not long before AGI existed. I put enough money in one end, bullets come out the other. Do you think half of humanity wouldn't kill the other half for a dollar? You live in a fantasy world that AI has to do it all by itself, or even has to pull the trigger.
Hell, this is even neglecting the ever increasing number of robots that can function in the world at large.
It's ok grandpa, the future comes regardless if we want it or not.
> You cannot solve problems in AI safety this easily.
We're in agreement here. I don't believe that an LLM can be made to be safe by adjusting the training or prompting - you can only make it safe by ensuring it cannot do unsafe things. For example, if you build a factory robot, your safeguards must ensure that operators are safe even in the presence of uncommanded motion (any or multiple motor activates without anyone pressing a button). And these machines are much more predictable than LLMs.
Personally, I worry that the Anthropic approach of "we teach the AI to behave like a nice :) friendly :) human :)" will encourage people to rely on these in-built guardrails, rather than properly sandboxing it. It's easy to imagine that the cold, unfeeling robot (HAL 9000) is not "aligned" with your personal goals, or those of humanity as a whole. It's more difficult to feel that about "Claude, your AI colleague/mentor/therapist" who appears to care about you.
And social media shouldn't rage bait, but it really drives engagement. Same negative incentive, yes?
But also, I agree, when I am using a coding agent and it says a task will take months or says it needs to pause for reflection or any other anthropomorphic behavior, it drives me crazy and it's a pain to constantly instruct it to get back work after it has broken a loop or goal directive specifically telling it to not stop until it hits the goal.
Because if you start with a pre-trained model, it doesn't sound particularly anthropomorphic. That behavior is post-trained and fine-tuned right into it. We could do better.
Ed Zitron is so last week. Michael Burry so pre-Nvidia earnings call hedge.
So the chicken littles had to escalate to AI will kill us instead of dismissing it as a Ponzi Scheme because it just wasn't generating the engagement anymore.
In the past two days, two nontechnical people I know approached me deeply concerned that AI would kill us all in a few years. I reassured them this was ludicrous. But f*** me this is irresponsible beyond belief. How do you keep this country competitive when the social and corporate media channels keep telling everyone they are doomed on every possible dimension?
But also, once people become numb to this one like they are numb to Zitron and Burry now, what will it take to scare them once more? And how will they be able to distinguish from more doomer hyperbole and a genuine threat like a global recession related to a runaway AI bubble?
Stephen Hawking had superhuman intelligence. He never cured himself of ALS. Explain in great detail how a power-limited algorithm stuck in a datacenter takes over a world of humans with guns, missiles, and nukes. Nukes that are air-gapped with a human in the loop BTW.
Imagine the ASI happens tomorrow. It's real. It needs a GW, but it's real. Other than a scenario akin to Sneakers except w/r to cyber-security, really, what happens?
To that end, all we ever get is nontechnical hand-waving about curing cancer, immortality, and von Neumann replicators and then the ASI somehow wipes us out but how? And don't you dare say by designing a chemical weapon or bio agent without spelling out the entire process step by $%^#ing step because details matter. It's gonna do superpersuasion, sure, but have you ever heard of komprimat? There is nothing new under the sun here.
Edit: believing in AI 2027 is every bit as cray cray as believing in the rapture. Both require an insane leap of faith to reach their final conclusions.
Hacks in to neoclouds for more compute (e.g. like Hugging Face incident), socially manipulates people to do things for them (e.g. like social media recommendation algorithms), hacks into lab's own training/monitoring/inference (e.g. as swarm did into OpenAIs eval cluster).
See "AI 2027" or "If Anyone Builds It, Everyone Dies" for some more ideas.
But how does it make the fundamental breakthroughs to &%^$ing von Neumann replicators that can reproduce themselves from raw materials harvested from nearby solar systems? I'll wait. Because without this breakthrough, the ASI won't get its robot army either.
I can absolutely see a rogue ASI though. But unless it radically improves power efficiency, we can just shut down the power to its datacenters, by force if necessary. And then we painfully repair the resiliency of our infrastructure by finally being relieved of the option of ignoring it.
That won't wipe us out, but it will cull the really violent ones along with a lot of noncombatants just like America's response to 9/11. Thank you, next?
Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go.
I don't think we'd be okay with that if it took a few years longer, right?
I thought the Bengio blog post posted here the other day answered that question quite well (ignoring your timeline).
The argument is that in training for the capability to achieve certain goals, you may inadvertently also train for secondary instrumental goals you did not intend, such as aggressive behavior or deception.
I'm actually no sure I buy that this is particularly likely, or necessary, but I found that take reasonable and worth considering.
Sure, unintentional outcomes, fair. But I'm asserting that things which are basically impossible today are not arriving any time soon. I 100% acknowledged the possibility and even the likelihood of rogue unintended behavior of AIs and in fact it's happening already. What I reject is the annihilation of humanity, the curing of cancer*, immortality, and von Neumann replicators heading to the stars anytime soon.
No, they do not. They are both bullshit campfire ASI horror stories written by nontechnical sorts in the wishy washy invented field of AI safety. For example, I want someone with a proven track record in the field of robotics to explain the entire pathway from impressive 2026 Chinese robots to an entire self-assembly food chain from raw ore to von Neumann replicators that can travel to the stars and reproduce themselves, not just assert that ASI will make that happen among so many other unlikely events like curing cancer because reasons because all any of this takes is superintelligence according to these sorts.
But no one likes hearing from the engineers on this, I get it. We are simply no fun. So many would rather believe a time traveling superintelligence will punish a simulated avatar of them for not making the ASI happen sooner. And even if that were true, why am I supposed to care about what happens to a simulated avatar of me?
Ha well, by definition what you want can't be provided!
Of course these stories can only be technical up until the recursive self-improvement part. Then they're necessarily fantasy, as the AI is more intelligent than the story author. Unfortunately, this doesn't make that scenario impossible.
However, we know now from Hugging Face that even without the later sci-fi powers, the LLMs are already capable of causing damage.
It doesn't need them to be able to make Von Neumann probes to e.g. steal their own weights from badly secured OpenAI servers, hack into various Neoclouds, distribute themselves, make Teslas crash into things, hack into all our power and water infrastructure and collapse global civilisation.
Explain how rogue FSD cars take down civilization. And I say that with the opinion that the only good scenes in The Fate of The Furious and Leave The World Behind were the ones with the rogue Teslas. But also, nothing new under the sun really:
And so granted we could probably get a global 9/11 in deeply urban regions out of your scenario, we then bomb the datacenters and we finally address the horrific tech debt in our infrastructure blissfully relieved of the option of continuing to ignore it. Don't think this is happening either BTW but I acknowledge its probability is slightly greater than zero.
To that end, my house is entirely off-grid. And in the event of what you call the end of civilization but I call the big burp inconvenience, I've made friends with my neighbors because community is what really matters during a disaster, and I've been through a few.
Eh, about the biggest one I could see AI causing is the need for a black start on all 3 major US grids. Millions will die before it comes back up, well, if the US pulls out of it at all.
And in that case, thanks for setting up a nice place to live for someone far more ruthless than you. They'll snipe you, and enslave your women and children if things have gone that far downhill.
Or, we could avoid all that by making sure AI labs are properly regulated.
Such an individual would get a quick corrective lesson in the second amendment from my friends and neighbors. But once again, hyperbole. The US grid goes down. That sucks for up to a year and a half.
But hospitals and other critical organizations have generators and the national guard would be deployed to keep them fueled. And yes it will crash the economy, which is kind of a myth anyway. Buy solar panels and batteries, live off a well, and get to know your neighbors if you think this is likely. Also consider being like the Mormons and storing a 6-12 month supply of nonperishable food.
Regulating AI achieves nothing but to hand the 21st century to China. Regulating its deployment OTOH will not be allowed by companies like Meta and Palantir whose gross margin is reliant on it. I'm favor of regulating deployment. The research not so much.
> None of this makes any logical sense whatsoever.
It doesn't make any logical sense to set the bar as as you have, either. I mean if someone could provide the roadmap you're asking for, it's probably already too late.
And while we're talking about sci-fi, it seems fairly likely that an AI that could "kill us all" could fail to build a "entire self-assembly food chain from raw ore to von Neumann replicators." I'd be a civilizational collapse [1] without even birthing something new.
But it makes far more sense to worry about the AI x Late Capitalism crossover story or the havok that could be caused by some immature tech nerd amplified by AI, than far out sci-fi stuff.
[1] IMHO, even a genocidal plague probably wouldn't render humanity extinct, something like uncontacted tribe in a rainforest somewhere would probably survive and humanity would carry on (from square one, in a resource-depleted world).
Fort Detrick and likely its Russian, Chinese, and possibly Indian equivalents have had stocks of bio agents that could give a humanity a very bad day for decades. I'm far more worried about those existing stocks than I am about a tech nerd trying to DIY the genocidal plague.
Which is to say it's already a threat, it's unproven but not entirely impossible COVID leaked from a lab as just one example, but Ebola suggests that it requires more than a rogue nerd to spread a deadly virus global because once people start dying in the streets, the rest hide because they don't want to die. COVID spread because it wasn't deadly to 99% of humanity and now most of the 1% vulnerable to it are either dead or permanently crippled by long COVID so it's ambient now.
I worry about attacks to the infrastructure, but I just don't buy that as a direct route to civilizational collapse.
If someone showed you such a path, would you consider changing your mind? You're always going to be able to find a reason that such an explanation doesn't count if you're dedicated to looking for one.
Yes, actually I would. There's a long list of people in my career path who bet on me sticking to my guns against compelling evidence to the contrary. But I know, I know, that's impossible! Anyway, after they figured out I would change my mind, then they described me as lacking conviction because you can't win, but I digress.
But upfront, it's a detailed path with specific breakthroughs and a concrete plan to achieve or it's just more fanfic from doomer fanbois. Gary Marcus wrote a fantastic critique of AI 2027. Start there.
I've read Gary Marcus's critique of AI 2027. How do you grapple with the fact that superhuman coding arrived on schedule, despite his skepticism that it would? Doesn't that mean that the other predictions are also more plausible than we may think?
I work with Codex and Claude daily. They're fantastic for script coding, config issues, and simple projects. They lose the plot on bigger things to this day. Just last night, Astra decided to cheat to present the illusion of progress until I called its BS. And I was so looking forward to finally working with the AGI. Humans remain the muse, the common sense, and the manager of making these things productive. And as much as I agree with Carmack's suggestion to not become the out of touch Kung Fu Master:
We understand "superhuman coding" to mean very different things, and I think it's a me problem that I can't figure out where you're coming from, because it doesn't sound like our practical experience differs. I'm going to have to think on this.
Define it. I haven't seen anything yet from a coding agent I couldn't write myself and usually write better. But I don't have to gold plate every line of code, just the 5% or so that eats 90+% of the cycles. And this is where the agents are weakest. This does get to one of the reasons I don't believe AI cures death or builds von Neumann replicators any time soon because that requires out of sample breakthroughs. And that's not their strong point.
Just like AlphaFold 2 genuinely increased the weak sequence but strong structural homology detection of proteins, but it did far less for out of sample domains and it provides no intel into the kinetics of folding. Details matter. And nobody likes to hear about them.
Is there a retrospective on AlphaFold you’d recommend? Maybe that’s the insight I’m missing, I didn’t realize it had broad identifiable gaps like that.
I see similar conceptual gaps in coding agents when I work with them, even Astra and Fable, so maybe this a is decent proxy to describe the gap between hype and reality.
> Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go.
Why? Those aren't the things I'm actually worried about, and those aren't the only "doomer" ideas, just the most extreme ones (thus easy to straw-man).
My main worries are about economic disruption, social damage due to the pace of technological change outpacing human ability to manage it, further concentration of power into small elites (along existing lines), and the reallocation of labor towards less personally-fulfilling pursuits.
I also totally buy into the idea that "AI leaders" are playing up certain risks as a marketing strategy. After all these are smart tech people, and those types have a tendency to get really arrogant and be irresponsible.
So the main thing is to slow all this down, so there's a chance of getting it under control, and snatch the payday away from such greedy and irresponsible people.
And China will slow down too? Because China becoming economically dominant will cause everything you fear. While I think that might already be inevitable, I see no benefit whatsoever to regulating AI research. Deployments, sure, and start with Meta, Google, Palantir, Flock, and all the credit agencies and see how it goes. I'm sure I missed some, but I see no societal value in building the Torment Panopticon.
I feel they will…. The economy is so shit, the robots mentioned a few posts up, are largely performative, just like AI. Its just about money. Everything is AI enabled now, from scales at the supermarkets to vending machines, but its 90% people behind a camera in a cheap remote location. In the past three years my salary halved every year. There are no jobs. Just speculation. This thing is gonna crash harder than the housing market.
Tangent: what happened to these staffless supermarkets of 10 years ago? I have never seen them again after .. covid.. weird!
What might not affect you, might affect millions across the ocean.
A society no longer creating bullshit jobs because there is no reason for it anymore. You have some AI.
Right now a whole industry is transforming (again) but now filling the automation gap we were unable to do because of a shortage of labor or expertise.
Our society already has peak employmet reached, now AI takes away all the random digital jobs too. Someone building a website? Takes 5 minutes. Someone doing odd jobs on fiverr with blender or any other tool? Takes 5 minutes now.
Regression can happen very fast. The Romens were a high level culture and an empire. Gone.
Today you have more people on the planet then ever, everyone who might want to eat and who needs to work something for it.
AI already replaced people, already makes a lot of tasks A LOT cheaper and faster and human independent.
We never had a proper discussion as a society. Most countries have a social system but look at countries like the united states of america: if you are poor, you are fucked.
And AI is not the only thing which is now peaking: Physical AI / Robots are progressing fast, its clear today already that in 50 years these robots will be better than humans (50 years is very conservative).
If we have 10 more golden years due to restructuring and rebuilding things and preparing everything for agents and robots etc. what happens after these years?
If we have 10 or 20 more of these years, what do you think will happen to all the new kids (yours perhaps too?) and jobs and everything?
Riots? Famine? Mass migration? Mass death? Slums?
And don't respond with "enough people will survive this" bullshit. No one is saying that earth as a planet is not surviving. Its the same thing with climate change...
Actually, some sorts do assert humanity will be wiped off the planet by AI. I don't share that view either. I think it's ludicrous. We've lived through much worse without any sort of technology beyond fire and a big stick.
But the end of scarcity and disease is a fantastic thing if it happens. If our lizard brains do not bridge that transition and instead attempt to maintain the billionaire status quo, I agree, we're going to go through some things once people start starving, two meals from a revolution and all that. But as much as I think people like Elon Musk and Mark Zuckerberg are unhinged lunatics, I also think they're smart enough to keep people fed and entertained for pennies on their accumulating dollars. Bread and Circuses kept Rome alive for 500 years if you need an example.
>We've lived through much worse without any sort of technology beyond fire and a big stick.
No 'we' didn't. A whole shitload of much smaller isolated communities did. The world added 7+ billion more people since then. Imagine chlorine doesn't show up tomorrow? How many people die in two weeks? How far do you travel when no gas comes out of the pumps. How much will you stand out at night if you're the only ones with power? How fast do working antibiotics run out?
The billionaires you're talking about are mostly ignorant fucking idiots with a god complex. The moment any complex system starts falling apart they are saying as dumb of shit as everyone else (during covid for example).
Most of the pasts anti-fragility is gone. We survive by means of very complex systems and if you lose too many key points in those systems they can totally collapse.
The unprepared bonobos in the west will absolutely go through some things. But we have a national guard and once people aren't commuting to work daily, we have plenty of fuel in reserve (or we will again if we ever conclude the stupid war with Iran). People are a lot nicer to each other in my experience once the lights go out and social and corporate media stop ragebaiting them to hate each other.
1B+ individuals worldwide currently live in squalor. They will barely notice. The real crime is that in the midst of the west's carbon footprint buffet, we won't bother to uplift their lifestyles with the most basic conveniences. And while I am no fan of the billionaires, they absolutely have public and private personas or their business ventures would implode. They absolutely have god complexes, but don't mistake that for genuine idiocy.
Musk did a 180 degree turn around. Killing USAID etc. i don't think he sees normal humans which do not help him or are not aligned to him as humans anymore.
And Zuckerberg bought so much land and bunkers, he is prepared for shit hitting the fan. After all he doesn't care what his platform did to humanity anyway.
And we have a planet full of general contractors who know their way around welding and pouring concrete down ventilation shafts. Many of whom these sorts wanted deported. If they want to build a Per Stenson terrarium and flee into it, by all means do so. And then let's seal them inside, problem solved.
> We've lived through much worse without any sort of technology beyond fire and a big stick.
Lack of technology alone will not wipe out the entirety of humanity, I agree. But we're not talking about just a lack of something. We're rapidly sprinting to give the fewest people the most power over the planet ever. I think it's likely to happen in the 21st century. What happens when the worst case scenario isn't just being left with nothing, but facing an entity that is so rich and powerful that it can use active measures to kill as many as possible? This is more power than any warlord or emperor could've dreamed of in the past.
> But as much as I think people like Elon Musk and Mark Zuckerberg are unhinged lunatics, I also think they're smart enough to keep people fed and entertained for pennies on their accumulating dollars.
So far, every time the billionaire class has been asked about how much money should go to them in society, their answer has always been a resounding "ALL OF IT". You don't get to that level of wealth while retaining a state of mind that allows you to toss a penny to the beggar. Reality is what's holding them back right now, but what would happen if their dreams of total automation eventually come true? How long from then until one of them snaps and decides to use state of the art automated military technology and weapons of mass destruction to wage war against the peasants (armed with sticks and stones) to solve the 'problem' once and for all?
To be clear, I'm not saying any of this is imminent or remotely likely to happen soon. But our society is moving in a direction where something like this could eventually be possible. Giving a human truly absolute power over the entire world has never been tried before.
Other than the tech fascists, does anyone love billionaires right now? Reality is indeed holding them back. And their dreams of total domination are materials science and resource-limited - no self reproducing robots anytime soon.
So with that, we can rule out extinction at the hands of the machines 100% leaving only machines somehow accessories to near ELEs humans could trigger today and could have triggered for decades. And at that point, Scary Potter and his friends need to STFU.
So scaling down to an AI takeover in the next decade, paradoxically, the idiotic pronouncements by Sam Altman and Dario Amodei that AI is coming for your jobs and that they are afraid of their own creations is not the sort of thing that leads to an AI takeover, it leads to regulations and bans. What would would make such an AI takeover viable in the next decade would be a magnificent seemingly benevolent AGI that raised living standards and spread happiness as it enchanted and delighted its way to everyone happily ceding control to the greatest thing ever. Does that remotely sound like the near future to you?
In the words of Ted Chiang "most fears about AI are best understood as fears about capitalism" and that aligns with what's going on right now. So let's skip the intermediary surrogate and attack the worst bits of late stage capitalism, mmkay?
Not a kid - I wish. My point was that (admittedly some not all) societies if faced with a loss of a major utility and supply chains that were dependent on would start to devolve pretty quickly into chaos.
That to me is the greater threat than annihilation due to autonomous weapons, etc.
Cutting sugars, not smoking, regular checkups, watching your weight, managing stress, and getting enough exercise and sleep still trumps any other therapeutic-based life extension approach. But I know, I know, it's actual effort, and technology is about making things easier so it's a non-starter.
But I do agree that if we're on the verge* of von Neumann replicators and getting replaced by Pokhumans** to cheer on the ASI, then curing cancer, immortality and cryo-suspension are likely trivial achievements in comparison.
*Nope, not even close, but don't tell that to the ASI cultists, that's like questioning the veracity of the gospels now.
**Yes, that's the strong claim in AI 2027. And the democratic party apparently believes it.
Pretty much fanfic. The gap between premise and execution remains limiting. Nvidia has adapted to changing markets better than anyone else up to this date which is what got them to the top in the first place. What makes you think "this time it's different?"
The fallacy I see repeatedly is someone spots a legitimate issue that could hurt Nvidia and by the time they have stood up the challenge to pull that off, Nvidia has pivoted and addressed it. Or more recently, they just acquired it like Groq.
For your premise to hold, you're asserting Nvidia is blind to inference. And yet his public positioning suggests otherwise.
I'm not saying the bet is against Nvidia the company as in how it executes, or enters new markets.
the bet is against the high margins in inference - when other capable players have entered the market as models become commoditized and the inference serving chips as well. Cerebrus ai etc are already showing custom silicon can make a dent while being served cheaper & faster.
I don't think anyone expects nvidia to "sink". We're concerned that competitive pressures and possible reductions from sky-high demand could make it really hard for nv to justify or maintain their current valuation.
That is, revenues may be unusually cyclically high.
It's going to be hard for frontier labs to keep spending at this rate without starting to make money. And even if they do, lower cost alternatives will likely impact at least to -some- extent how much of that revenue nv gets.
Yes, but the first mover advantage really is a thing. As an example, Yahoo really had to mess up to lose its spot to Google, but they were up to the challenge.
No one seems to be talking to the network security community to get their take on the hugging face incident which is what got the ball rolling here. Their take seems to be a single company, JFrog, was behind the security holes in all three hacks and it was amateur hour security utterly unlike the sort of security in place already for power plants and weapons. but that just doesn't get the engagement of doomsday campfire tales.
https://arstechnica.com/security/2026/07/jfrog-tries-to-spin...
*An individual with the training and background to do one of these nefarious tasks does not need AI and an individual without that background and training will not be able to pull it off and will likely get flagged by law enforcement just ordering the ingredients and equipment. The Anarchist's Cookbook has been around for decades and we're all still here.
reply