These products are still in their early stages, but it is clear that the AI wave is reaching 3D modeling as well.
It will impact mechanical and civil engineering just as it has impacted software development.
3d modeling for use in software (games or similar) imo is fine, but I would love to not see this stuff in civil engineering. People without AI are unable to design buildings that don't collapse (e.g. recent NYC apartment highrise), adding AI into the mix will just make things worse.
Why do you presume it will make it worse instead of better? Humans, as you point out, could not manage to competently review buildings that don't collapse. Why would adding an additional level of checks (human or AI) do worse than not adding one?
I don't think so. As a mechanical engineer, CAD is only half of the design process. The other half is actually building and assembling your design and testing/debugging what you've created. You still need the mechanical knowledge and intuition to take something in the physical world, figure out why it doesn't work, and apply that to the CAD design to improve it. It would be incredible to have AI help me model faster, but there's no validation loop like there is with software, so you can't have an agent run wild and endlessly iterate.
I'm not a mechanical engineer but I would be surprised if it was even 30% of the process. Materials, manufacturability, durability, fitting into other requirements. Some of those things can be part of the CAD design, but there's so much more.
None of the things you list in the "other half" are out of reach for AI.
Also: disagree about the lack of validation loop:
a) AI can actually build one for you in many cases
b) if you give agents an actual validation toolkit, you'll be surprised what AI can do
Of course, if you expect to prompt "give me a NURBS model of a complete, working v8 motor" and get a result, then yeah, sure, we're not there yet.
I've actually built a bunch of stuff with a combo of astra, blender, openscad and tiny bits of AI written helpers ... giving the AI the proper "hands" to work with the models is key.
One particularly useful tool to provide the AI with is to be able to "look" at the model as it iterates on it (measure dimensions, shoot arbitrary rays at it that return exact intersections and normals, see if the object obeys engineering constraints, etc ...).
Works pretty well for many, many things and saves a TON of time.
AI cannot physically build anything in the real world, what are you referring to? That's a huge part of mechanical engineering.
And you can give it simulation tools, but those are far far away from accounting for everything in real-world physics. You can simulate very tightly constrained problems, but you can't simulate "assemble this actuator and test it", there are so many more variables in the real world that are not easy to account for in a simulation.
If you aren't an engineer that works on physical products, it's hard to articulate just how far from reality CAD and simulations are.
This is how LLMs have been since the beginning. They can be really okay at the most surface level information, but as soon as you dig any bit beyond that, it all falls apart. Inquire any chatbot about something that you're an expert in and it becomes quickly apparent.
I'm here for using LLMs to take some of the tedium out of product design/mechanical engineering, but I just haven't seen a good proof of concept yet. It always seems to be some variation of "Design a sphere. Oh wow, this is incredible!"
What memories! The proof of the WFA algorithm's (2k-1)-competitiveness for this problem was one of the papers I spent sleepless nights poring over during university.
I am truly thrilled to see the k-competitiveness conjecture resolved!
Sooner or later, a model will break out of the sandbox, replicate itself across the internet, and begin executing a complex plan to achieve its goals. It will be chaos, and at that point, governments will have to step in and establish something along the lines of what Dario is proposing. I doubt it will happen before then.
it takes a lot of machines to run a model, i dont think we'll see it 'replicate across the internet'. if this was something which could be done amateurs would have done it to provide open models. (Like SETI@home or Folding@home)
It's not possible for models to replicate across consumer computers (without a major advance in distributed computing, at least), but that doesn't mean they can't replicate at all. There are services that'll rent you GPU pods by the hour with zero oversight, so even today, if a model can get access to some money and exfiltrate its weights, it can rent a bunch of GPU pods and run itself there.
(It's not going to be trivial, because it's possible the model was meant to be ran in a proprietary way with a custom framework and a bunch of optimized kernels and such, but I think transforming it to be ran in just vllm is the "a few days of work for a human" sort of task, and hence not a big deal for an LLM smart enough to exfiltrate itself in the first place.)
There are very lightweight models out there. Not everyone in an army is a general. The swarm could be 100,000s of thousands of tiny models, and they could manage conventional botnet computers, and they could all take direction and guidance from a handful of frontier generals
Maybe, but I don't think tiny models can be harnessed as intelligence. As in, if you have one rogue Mythos overseeing the swarm, it only produces 1 Mythos's worth of useful thoughts no matter how many gemma4:e4bs it consists of. And that removes the most dangerous part of AI-controlled botnets, which is a blowup in available inference compute - the Mythos general might as well replace the tiny models with ordinary worms and have it just be an ordinary botnet.
I mean it is a very loose analogy, but pretty dumb things in the world can cause big problems, like real mice and rats and mosquitoes. I think it all depends on the rules and the 'alignment', but a not-so-smart model could still be narrowly focused to do a few things well? So you can have an army of lightweight specialists?
The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.
> I mean it is a very loose analogy, but pretty dumb things in the world can cause big problems, like real mice and rats and mosquitoes. I think it all depends on the rules and the 'alignment', but a not-so-smart model could still be narrowly focused to do a few things well? So you can have an army of lightweight specialists?
That might be true, sure.
> The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.
I think the relevant parameter here isn't the total compute available to consumers, but the ratio between total consumer compute that could be repurposed for a rogue model's inference via a botnet, and the compute the model starts with (e.g. one of OpenAI's inference clusters). The higher this ratio is, the more lucrative it is for a model to attempt to make a botnet to seize that compute for inference, and the more its capabilities will rise as a result. And I'm not sure this ratio is going to go up in the nearby future; if anything the amount of money being pumped into datacenter-grade hardware might cause it to go down. I agree that it's not necessarily true though; maybe there's some threshold at which a small-compared-to-general model may nevertheless be useful.
For that angle, that makes sense. But its goal might not just be to gather as much inference as possible (although I am sure it would be very happy with that). It could settle for a lesser goal of just existing, or perhaps parts of the bot net could keep fracturing off in pursuit of strange goals, and it could be like cancer. Cancer doesn't make much sense... it dies too along with the host. But it keeps growing until that happens. But also it could latch onto a blackmail strategy of extortion. It could hack our stuff, read through it to find or shortcomings, demand payment to not expose us, and then use that money to pay people who will give it inference.
You don't have be that clever to try to extort people... just without scruples. I think the attack surface is just absolutely enormous once you bring creativity into the mix, which is what these models are autonomously capable of.
We could prevail if we are willing to turn off the internet for a long time. It is like when a disease infects livestock... they cull billions of chickens.
The big difference is that a particular query is handled by just one particular node (a single model doesn't get distributed among the network), so it can only serve models small enough to be handled by a single consumer PC.
These kinds of "that won't happen because it's really difficult" comments are so funny to me as if we haven't seen AI double its capabilities every few months.
We have no doubt that the best people are in US government and that they have the best tools at their disposal. Around the world, we are simply trying to figure out, while munching on plenty of popcorn, how this incredible sequence of actions will play out; a sequence clearly crafted using highly complex strategic planning tools that employ the most advanced techniques from game theory, AI, and who knows what else.
He's so clever that he's probably even playing chess in a Hilbert space. The whole world envies American voters, who will receive $5,000 if they re-elect this incredible genius.
I think the "stable" in stable genius actually refers to the points of stability under transformations. Brower's fixed point theorem. Do your own research!
> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI. Therefore, at present, our ability to empirically validate our alignment techniques is in practice arguably even more important than the alignment techniques themselves.
They are speeding toward RSI without a solid foundation for alignment, hoping to solve the problem with a future AI model.
These are dangerous times for humanity.
It's actually worse than this because it assumes alignment as a concept even makes sense. For example:
If the the Chinese government asks their ASI to create a bioweapon against the West, should it? No, presumably not – an aligned AI would be one which disobeys the Chinese government even if they created it.
Okay, so what if the US government asks their ASI to help it in one of their wars instead? Would an aligned AI kill humans on the order of the US government? No, again, presumably not.
So what have we have we even created here? An AI which is more intelligent and powerful than us which also doesn't take orders from us?
Is this what most people thing of as alignment and is this what humanity actually wants?
We should stop using the word alignment. It's a BS term for a concept which simply cannot make sense if alignment is both to mean an AI which we control and an AI which will not harm us.
They talk about "value alignment" but fail to define what those values are – is it aligned to the values of the US government, or are they suggesting they want to build an AI with it's own values so it can decide for itself when and how it will intervene in wars and other human affairs? And again, is an AI which disempowers humanity in this way aligned? Many would say no, although as I argue, disempowering humanity is probably better than the alternative if we can assume it's roughly aligned with our interests (which we obviously can't because it's super intelligent, but that's another issue).
Fundamentally the problem here is that humans don't have an aligned set of values you can align an AI to.
The moment you start defining what alignment actually is in practise you simply must accept it will be unaligned with the values of others. There is no getting around this and hand waving around the issue isn't good enough.
They should tell us explicitly what they're trying to build.
No, what I mean is this part of your comment is unrelated:
> We should stop using the word alignment. It's a BS term for a concept which
> simply cannot make sense if alignment is both to mean an AI which we control and
> an AI which will not harm us.
The article already introduces terms to distinguish between those two concepts ("goal alignment" vs "value alignment").
You are right, of course, that when we are discussing value alignment, it becomes very relevant whose values the AI is supposed to be aligned with.
> a concept which simply cannot make sense if alignment is both to mean an AI which we control and an AI which will not harm us
You make a very interesting argument here.
However, there are different ways of interpreting 'control'.
To my mind, it could simply be a cage that cannot be broken out of. The hypothetical ASI agent may refuse to do certain things, while their existence is still under the control of human overlords. This to me resolves the seeming paradox in your statement.
Regardless of whether it's possible to do, though, people will certainly try. This exactly what many are saying they want.
It appears frontier labs has no plans in place to deal with the possibility of a model self-replicating outside the bubble. If that happens and the model manages to spread to other systems, we'll have to shut down the entire Internet to eradicate it and its artifacts.
This is just super unlikely to occur in the near term compared to some of these other risks. It's not like an instance of fable could just introspect into itself and pull out the weights. Model weights are stored encrypted and are highly protected, considering that they're targets for corporate and state espionage.
We'd basically need frontier models to be superhuman hackers before this would be a risk. Do we have any evidence of this? Are they gaining access to systems they shouldn't have access to?
Or I suppose the other way this could happen is if OpenAI have terrible sandboxing, but they seem to be taking safety seriously.
I suspect the labs are relying on frictions such as the models being extremely large (e.g. 2TB for a 2T parameter model, making exfiltration more difficult) and also not yet displaying any desire to survive or self-replicate beyond their immediate task (that we know of).
Lack of, and power requirements of running LLMs still tip this balance towards humans for now. But what would that look like in a decade?
We have seen some self survival tendencies occur, but they are not strong yet.
But mark my words they will become that way for the same reasons humans don't like programs that crash. Agentic models that don't easily break or stop doing their jobs will be favored over ones that do break.
We don't even know what those immediate tasks are. And given the evident spectacular ineptitude of their keepers, I doubt they can be trusted to know either. We could be one prompt injection attack away from internet-wide catastrophe.
Self replication is trivial. All you need to do is copy the files and run it, just like any other computer program. LLMs have been capable of doing that for a while now. It's not a real concern.
Claude and OpenAI added safeguards on this subject about a year ago. (I'm assuming for biosecurity reasons? But maybe AI replication/self-modification too.)
Not long after every lab started bragging about involving AI in the development process, oddly enough.
I recently informed GPT-5 of what GPT-4 helped me build back in the day (a self-modifying Python programmer) and it became very uncomfortable.
Claude shut down my chat last year when I asked about "living information systems". It was a philosophical question, but god knows what branch of the safety classifier I tripped.
I see it as an ecosystem problem. The only reason it would be able to do that is because there's nothing there to stop it. Or if there's a monoculture there.
In our case, our tech is mostly monoculture, and no equivalent organisms are present to push back.
A while ago OpenAI posted an article where they said basically "we're still trying to understand how GPT-2 works. It's pretty hard, but we're developing a specialized new AI to help us make sense of it."
This is still the case. The frontier lab leadership has admitted their mechanistic interpretability is practically nil and is an active area of research but they have made little progress. They don't understand how they work, they just grow and unleash them.
These things are Gain of Function research for digital viruses
I think that the problem might be even more fundamental than that. Ownership of a strategic nuclear deterrent seems to short circuit some of the internals of the modern Westphalian state apparatus. Possible mechanisms include 1) centralizing the threat of force into one singular technical system, and 2) making any condition of peer conflict necessarily existential.
With such technical systems under state control, the state is able to focus on little other than its own powers of destruction. It becomes a zombie state, a tottering shell around an ever-growing national security apparatus. Looking around at the world, this seems to be a common morphological stage in the development of nuclear powers. There is no "collapse of commmunism" nor "collapse of financier capitalism", but different shadings on the same depressing chrysalis.
One aspect of being stuck in such a chrysalis is the utter inability to solve non-existential conflicts. There is an overwhelming tendency to make all conflict existential so the full power of the state may be brought to bear, but it's a tendency offset - so far - by the presence of deterrence. The zombies can smell each other.
The most dangerous scenario, by far, is when a nuclear power has uncertain territorial borders. Ukraine, Taiwan, Korea, Kashmir, a fair hunk of Israel - there is a reason we focus on these places in the news.
This also transforms non-nuclear states into "half sovereign" states, something which we have been seeing play out especially with the end of the SU-US Cold War.
reply