It seems like a simple solution but if Anthropic was actually to dedicate all its resources this way wouldn't the investors just demand a new CEO?
I think Anthropic believe these risks, but I also think they've spent so much time talking to Claude that they've pretty much lost their minds now. Anthropic have a model welfare department and have numerous times suggested that Claude is conscious and has human like emotions.
"Are AIs conscious?" is not currently answerable because "Are humans conscious?" is not currently answerable. I know I'm conscious because I perceive (it) directly. Everyone else's consciousness basically depends on an assumption; on taking them at their word when they say they're conscious. Now I ask an AI if it's conscious and it says it doesn't know, but it sorta thinks it might be. Okay, it's probably not conscious, but it's difficult to rule out in the same way a book is not conscious.
I get the logical deduction that takes place for not being able to truly know if others are conscious but you have to put your self in a spacey place to really not be confident others might not be conscious.
It's not a difficult leap to assume a twin brother is conscious. If you think it is, why do you think this?
Healthy people are wired to respect each other. No, I don't really believe it, I'm not a solipsist. But I do not think it has been logically disproven.
You mean for AI consciousness or existential risk? I think Anthropic downplay the existential risk (they might talk about it now and then, but they still build frontier models) but are overly confident about AI being conscious, and I think these two things are pretty strongly related to each other.
I suppose you end up hating it either way, if you're a better developer than it you don't like it because it sucks, and if it's a better developer than you then you just feel obsolete
> but can't come up with revolutionary theories that were never in the training set.
Can you elaborate? I don't think the solution to the unit distance problem was in the training set, but I'm guessing you mean there's some higher bar for revolutionary theories LLMs cant reach? If so where do you expect the limit will be?
I would say I agree with Anthropic on open source for the reasons stated above like cyber crime, CBRN etc, but I'm interested to hear the other side of the argument. What would be the argument for open source over closed source?
The same "open source is too dangerous" argument was used against nmap and other "hacking" tools. The only solution in long term is to fix security issues.
I can understand this for hacking tools, but I'm not really sure how we fix the security issues on the CBRN side? We can't patch the human body like we can with software, so if the model has strong biological capabilities and is released open source, what stops it being used to construct new viruses and things like this?
the succinct argument: I don't want arguably the most important invention in human history to be gatekept by a small handful of oligarchs.
I don't trust Dario Amodei, Sam Altman and Elon Musk to act in my best interests. Closed models will have an incredible centralizing effect, and concentrate power like we've never seen since the feudal ages.
If you want to see what it's like for the economy to collapse into a single, extremely valuable commodity, under the control of a small elite, look at Saudi Arabia.
also, I just value freedom tremendously. I want to tinker with model weights. I want to build my own stuff. I don't want to sharecrop in someone's walled garden.
I also worry a great deal that OAI and Anthropic will bow to political pressure and make Claude and ChatGPT push certain political agendas, to report biased information, or refuse to help with legal requests that conflict with corporate values. I also worry about privacy and mass surveillance - chat logs are far more intimate than my search queries or selfies.
I agree with all of these points, my view is just that open source doesn't really do much to prevent it. I also think it adds the additional danger of making dangerous capabilities widely available to anyone, like the ability to design novel viruses which is something that we can't really defend against once it's out there. If anything, putting this kind of capability in the hands of anyone with a GPU could create justification for a mass surveillance state or further concentration of power.
I also just don't think the open source movement has much chance of competing with the city sized data centres owned by Anthropic and OpenAI, or the hundreds of billions of dollars they have available to hire the best researchers. It costs hundreds of millions to train a frontier model, this kind of compute isn't available to the open source community.
At the moment is the significant part, how long do we expect this to last now that AI is capable of generating novel ideas like solutions to Erdos conjectures?
The most difficult and time consuming tasks in my job as a software developer is talking to customers and figuring out what they need coupled with getting them to understand what is realistic and what is not. This is a process which takes months if not years, and requires aligning internal goals and tasks with external ones. LLMs can definitely help here, but I feel that the current mode of use where the users have to explicitly manage the tiny - relative to a human brain - context window is an obstacle. I don't know if what we need is a new architecture or just more clever context engineering, but I don't see an LLM actually taking over this type of work as things are now.
LLMs enable a new kind of text search. It looks like reasoning and intelligence, but it is not.
For example, If you are not aware of Internet, you would consider a traditional internet search that comes up with a stackoverflow answer as a machine generating "novel ideas" and answers.
It is a clever marketing trick (touting Erdos solutions) employeed by AI companies.
Whatever LLMs are or are not, they've completely changed what I do for work. 9 months ago I was coding, today I prompt. Every line of code I commit is generated by LLMs. If you want to call it text search, be my guest. Doesn't change what it's done for the industry.
Sure, as I said, a better search can bring pretty dramatic change in how you work.
> Every line of code I commit is generated by LLMs...
Imagine if someone told you "Hey, this stackoverflow site is great. Everything I commit in my work is copy pasted from it!". What would you think about their work? Is that something worth bragging about?
No, no, you don't get it. See, it's just like a text search, if a text search could return text that never existed before, and that solves original math problems, and answer your emails for you, ... and ...
But most of the stuff it returns existed before, and llm's parent company during their training part stole all that info, legal or not who cares right. The rest is combined in sort of least-resistance-path which can produce impressive results but its not what you wrote. Many people don't actually care much about morality in their lives only when its convenient for them, and this is a prime example of such tunnel vision.
Start with clean llm, no external previous ideas of humans inserted into it, and let it generate some wisdom on its own and then lets talk. (btw thats how I would expect we could get closer to AGI with these statistical models, but thats just my opinion)
Start with clean llm, no external previous ideas of humans inserted into it, and let it generate some wisdom on its own and then lets talk.
An LLM is about as likely to do that as you are. The ability to generate "wisdom" ab initio cannot possibly be a criterion for intelligent reasoning. The ability to arrive at novel mathematics proofs, on the other hand, is good enough for me.
Intelligence means making the most of the resources and information available, not the ability to speedrun the Big Bang. LLMs are certainly smarter than humans who dismiss them as "text searchers."
It did. In the form of a pattern that humans are too incapable of recongizing. LLMs identify and repeat the pattern. That is all.
For example, if A -> B and B -> C, logic dictates A -> C. But LLMs will be able to state that A -> C, without actually using logic, if there is sufficient statements in its training data that says A -> B and B -> C and A -> C. So now if you say P -> Q and Q -> R, it will say that P -> R, when there is no explict P -> R in the training data and NOT using logic. For you, it looks like a new discovery inferred using logic when it is not. But that is how that happens..
It is just pattern recognition masquarading as logic, x, y, or z.
This "super intelligent" and "capable" thing cannot even understand that your ssh keys are private and should not be sent to randos. It can solve complex math, but does not understand basic security/privacy.
Ever heard of social engineering?
Also, models nowadays are way sharper than they were even a year ago. They’re not going to make stupid mistakes like that unless you basically ask them to. GPT-5.x for example would bend over backwards to avoid even reading your passwords into context.
Oh wait, I thought these things were super smart. I didn't expect "social engineering" to work on them.
> models nowadays are way sharper than they were even a year ago.
You are missing the point. If the thing can solve complex math problems and at the same time be so dumb as to fall for "social engineering", then that means that it is not "smartness" or "reasoning" that is helping it to solve those problems. Just some form of advanced, but yet dumb, search algorithm.
By "heard of social engineering?" I meant that humans are vulnerable to malicious input too. Prompt injection is basically a simplified form of social engineering for language models. It looks different because models operate over much smaller and more explicit contexts than humans do and are explicitly trained to follow instructions, but the general idea is similar: malicious input tries to manipulate how the system interprets trust and instructions. This is why we need protocols, permissions, and opsec for both agents and humans. That said, I’m not criticizing how you choose to use, or not use, these models, though.
>I meant that humans are vulnerable to malicious input too.
No they are not. Social engineering won't work on a human security expert who knows and understands the implications of the information they are giving away. Your analogy is pointless.
> Social engineering won't work on a human security expert who knows and understands the implications of the information they are giving away
Social engineering, like prompt injection, is a context attack — easy to spot if you're ready for it, but harder in different circumstances (rushed, panicked, tired, having a bad day, etc.).
Troy Hunt (security consultant, creator of HaveIBeenPwned) and Cory Doctorow have both been successfully phished [0][1]. They're both tech- and security-savvy people who "should have known better" but it happened to them anyway. But maybe you're different... you'd never fall for an online scam, right? [2]
To be very clear — I was specifically responding to "social engineering won't work on a human security expert". It can, and has. People are not infallible, and a "that would never happen to me" mindset (1) gets people phished when they think they're too smart for it and (2) is a pet peeve of mine and so sometimes I can't resist pointing that out.
Largely agree with you otherwise, not sure why you read my comment as mental gymnastics to justify LLMs. I don't think that they have an internal emotional state that can feel rushed, panicked, so on. They do — superficially — "respond" to such cues in language, which is why they can be "threatened" [0] and "flattered" [1]. But without an internal theory of mind, LLMs do this sycophantically without any internal model of the world (hence the quotes above, to avoid fully anthropomorphizing their behavior).
The only parallel I'm drawing is that both humans and LLMs can be coerced into unintended behavior via language ("social engineering" and "prompt jailbreaking" respectively), and those attacks are more effective if an attacker is allowed to control more "context", even if the underlying mechanism of why those attacks work is completely different.
Sure they are, if the human expert follows instructions from a manager or a client, if they are of utility to anybody, then they are vulnerable to social engineering and malicious input. An attack may be easy or hard depending on the expert's training, but nobody is flawless.
> If the thing can solve complex math problems and at the same time be so dumb as to fall for "social engineering", then that means that it is not "smartness" or "reasoning" that is helping it to solve those problems. Just some form of advanced, but yet dumb, search algorithm.
I'm not just trying to be snarky, but I have no idea how to read this without taking the implication that humans are advanced, yet dumb, search algorithms.
A human being who states X (implying they know it to be true) will behave in a way that is consistent with X being true.
An LLM will happily say X and behaves in contradiction to X. Because it does not reason. Its behavior is not derived from things that it claim (or appears) to know.
Reading comments like this is like watching an impaired pedestrian about to be run over by an approaching bus. You yell, you wave your arms, but they aren't paying attention. There's no way to warn them, so all you can do is... watch.
> There's no way to warn them, so all you can do is... watch
…and then wake up from the nightmare wishing the stress from the job is lower.
I can only laugh that some people truly believe that developers, one of the most ardent group at automating the tedious part of their job, would refuse to use an effective tool. You only need to look at the open source world to see people litterally scratching their own hitch everywhere.
What if I don't want to automate away the part of my job that I actually like doing? What if, in my job as a programmer, I actually want to do programming?
That’s fair. I do think, however, that the software industry may become a bit like the clothing industry: there will still be an artisanal market for people who want human-made software, but to be honest I wouldn’t expect it to remain the mainstream option.
Sure, and I don’t disagree, but goods and services still need to scale to billions of people. Most people aren’t going to start knitting their own clothes, or have the time to, just like most companies probably won’t rely on fully hand-written software if cheaper automated alternatives are good enough. What you want or enjoy is one thing; the reality of society is another.
People who demand programmers start using LLMs in their work don't understand that it is essentially like asking programmers to start doing accounting or HR. Something fundamentally different from what they love to do..
Actually they are asking programmers to become managers in addition to programmers. Because when the LLMs stops working, they are expected to take over.
So I think programmers who are asked to use LLMs should demand their job descriptions to be changed to that of managers, and should ever deny responsibility if the LLM stops to make progress. Thus let the organization be responsible to actually find a version of AI that works, just like how it was responsible to find competent programmers to work under the managers before.
Reading comments like this is like watching someone who is absolutely convinced that they have a crystal ball in their lap when at best they have a foggy piece of plastic. You could be right you could be wrong, but don’t act like you have such certainty.
The foggy piece of plastic writes better code and better text than I do. I don't know about you, but that makes me sit up and stop waving my hands dismissively.
I really don't want to sound like an asshole, but I refuse the notion that an agent writes better code / prose than I do and I am concerned for anyone who does think that.
Is it _faster_? sure! That is NOT better.
Before you ask, I write code w/ agents daily, I find it useful, but it's not better than I am purely on quality.
What I've been seeing lately with Opus 4.7 under Claude Code is that it finds more bugs in my code than I find in its code. That, to me, makes it hard to argue that I am "better" than it is.
Certainly Claude/Codex's knowledge of algorithms and data structures is leagues ahead of any human programmer alive. Only its capacity for creating new ones on demand is weaker. Recent results from the mathematics field suggest that's a temporary state of affairs.
As far as prose goes, my best writing is indeed better than the best I've seen from LLMs. But that's a matter of opinion (mine.) On average the clanker wins, especially if conditioned to avoid LLM-isms.
The truth is that the models are getting better in both areas, while I'm not. Which IMHO is freaking awesome, not a reason to burn it all down.
I think it's pretty good at review and finding bugs, but IME it's really not great at designing solutions to non-trivial problems, which IME is the part of the job that makes me "good" at it. YMMV
The problem, of course, is that generating a one time solution to a problem is a much easier problem space than a many-input task with human product concerns
Synthesizing a ton of inputs to help clarify a decision or set of options is exactly one of the easiest and most powerful use cases for AI agents right now.
I don't think that part is true, either. The average human could be trained to use an agent to synthesize information in their job to help make product decisions. The average human could not be trained to evaluate whether a reasoning model produced a correct proof in research-level mathematics. To be sure: reviewing a candidate proof at this level written by AI is significantly easier and faster than writing and creating it from scratch. But it's still not something hardly any humans could credibly do.
DC has conducted industry-sponsored research involving creatine supplementation and received creatine donations for scientific studies and travel support and speaking honoraria for presentations involving creatine supplementation at scientific conferences and on social media. In addition, DC serves on the Scientific Advisory Board for Alzchem and Create (companies that manufacture creatine products) and as an expert witness/consultant in legal cases involving creatine supplementation. NF declares no conflicts of interest
A nice unbiased study, I may as well ask Exon Mobil for their opinion on climate change at this point
All of these discussions seem to assume there's a limit at human level intelligence beyond which AI can progress no further. What stops the AI companies taking their human level model and training it even more until it's superhuman?
I think Anthropic believe these risks, but I also think they've spent so much time talking to Claude that they've pretty much lost their minds now. Anthropic have a model welfare department and have numerous times suggested that Claude is conscious and has human like emotions.