I have a friend who is one one the best programmers I know. He worked for an ad tech company more than a decade ago optimizing time to respond for personalized ad delivery. He was not proud of working to act on fingerprinting to optimize delivery but it was a challenging task and he framed his part as addressing the challenge not the ethics. People in our circle where like "good for you optimizing network delivery to 50ms!" even people who despised the idea of tracking users around the internet to serve them ads.
I'm sure most people who work for flock compartmentalize their jobs to "I work on fingerprinting vehicles" or "I work on advanced OCR", or "I work on creating camera to camera trajectory graphs". This way they don't need to think about (too often) the larger ramifications of what they are doing. When the press points out what is the reality and that most people are pissed at Flock then the employees likely have to confront what they do in a way that they never had to.
It is also reasonable for employees to hold the position that Flock is a public good and the vocal minority of people complaining about it are simply wrong.
That’s not a reasonable position to hold, unless and until Flock does much, much more when it comes to safeguards against abuse.
I can understand being pro-surveillance, but Flock is a case study in how not to implement surveillance systems. Believing that surveillance can be a public good can’t somehow turn Flock into a public good. It’s not.
This isn't an uncommon opinion among the public either. It's probably a minority, at least among those who are vocal about it, but it's not hard to find people who think they are beneficial.
I'm not sure why you're being downvoted. Perhaps it makes some people uncomfortable about their own jobs. In any case, it's not uncommon for people to moralize about something they're not doing, but given the opportunity to do that thing for some handsome reward (good pay, prestige, technically interesting work, etc), they easily rationalize their choice to participate in that thing.
This is basically what Google did when they pioneered the model of surveillance capitalism (see, e.g., The Age of Surveillance Capitalism).
Google simply provided an index on top of an existing library. Of course, a librarian has no value if he has no books to index over! But it's also worth noting that the Google "librarian" also leveraged the existing "social" structure of the internet: their core contribution (page rank) was a clever, efficient mechanism to extract the latent value in the pre-existing link structure of the internet. This structure (much like the pages themselves) had been curated by actual humans. Undoubtedly page rank was clever, but it was worthless without the existing websites (books) and the existing indexing information (the pre-existing, crowdsourced librarian work). Nonetheless, they successfully monetized it.
AI companies are even worse in the sense that initially Google was still sending traffic to the original webpages. (Until they didn't - https://www.eater.com/2017/9/12/16294380/yelp-google-scrapin...). So yes, the AI companies have even more thoroughly stolen the collective work of humanity than Google did.
so they basically have copies already of every webpage until they turned it off a few years ago (well they may still have it updated but not provide it as a service)
so it occurs to me they most definitely trained their "AI" on all that user cache
they may have even just turned it off as a service when they realized other "AI" could do the same thing
It turns out that the only way to make money online is ads.
Nobody is going to pay for ChatGPT. They'll just use the ad-infested version, like they do everything else online. Well some people will pay, but not enough to justify the insane amounts of money being poured into it by investors.
A bigger problem is that it will be impossible to know when you are seeing an ad: political groups (or the government, perhaps) will partner with openAI to subtly express different values.
It's still an advertisement, and the underlying marketplace is similar (pay for access to change behavior).
The best uses of AI will be surveillance, propaganda, cyberterrorism, and automated military tech.
The easiest off-the-shelf option would be a router running OpenWrt. IIRC, it natively uses dnsmasq, and the relevant blacklists can be obtained from here:
My own setup is DIY: a Debian box running Unbound (recursive DNS) with the RPZ blacklists from above. This gets rid of the upstream DNS service such as the ISP's completely, and prevents tampering or censorship.
For a home network network pihole or unbound also supports blocklists (bundled with opnsense for example if you also want a firewall). For Android, I use Rethink with Hagezi blocklists, so they block also when I am on mobile data (it is vpn based).
People think they're interacting with an "intelligence," when actually they're just getting a maximally optimized Weizenbaum feed. We're living through the sloppification of the human mind.
b) Some models released before the car wash problem was discovered would consistently get it right
c) Hardcoding it is pointless. No one is seriously asking that. It's just a trick question. Hardcoding one trick question won't fix its weakness at other simple trick questions.
d) Since it went viral on the internet, the next time they updated the knowledge cutoff, the LLM would likely be aware of the trick. It will fix itself without the labs doing anything special, even assuming the new models weren't smart enough to naturally figure it out.
Meh. The car wash problem is an underspecified statement. It's like, hey, I just popped into existence and someone asked me if they should drive to the car wash nearby.
It's not an insane assumption that the user isn't dumb and has some other reason to be asking the question other than it being a trick/stupid question (duh, if you want to wash your car you need to drive it to the car wash!). Taking it as some ultimate measure of intelligence simply doesn't make sense to me.
It turns out, oddly enough, that it's possible for it to be both. AI can simultaneously be used to destroy the commons with slop and also make contributions to new math (though isn't the jury still out on whether part of the idea was stolen from human mathematicians?)
In an embarrassing display, the manager over my arm of the company got told off by his manager for spamming him (and clients) with AI slop emails. And then was told by one of the regional managers for using Copilot generating a bunch of client-facing posters and images with phrasing that had not passed any sort of legal or basic fact checking.
The people going all in on it don't realize the downsides, limitations, or understand how they come off to other people with it all.
I think people who make being smart their whole identity are a out to use AI to turn the rest of the population into indentured servants.
The comparison between ELIZA and LLMs is valid you boil it down to "humans evolved for 6-7 million years, had spoken language for 500k years, but have only had something non-human that could generate convincingly novel language well enough to hold a conversation for a few decades".
There's no inherent reason it can't turn out having a non-human generate convincing enough language for conversation isn't a complete evolutionary blindspot the same way the short form feed has pretty much one-shotted society...
LLM chatbots are software designed to manipulate, addict, and mine data. Just like social media before it. But anyone who bothers to read the output in a domain they understand will discover they aren't all that no matter who OpenAI steals research from.
Or perhaps their wallets. There are a lot of programmers here. Some of them think that AI is the death of their profession, rather than a change in it.
"It is difficult to get a man to understand something when his salary depends upon his not understanding it." - Upton Sinclair
I was a programmer. AI is the evolution of my career. I'm in an interesting place. I've never been good at interviewing and I have certainly only gotten worse due to an admitted atrophy of thinking in "code." But I've been putting more effort in designing systems (agentic and otherwise) and building things at work as for fun.
It's been an absolute boon to finally build out all of the fun side projects I had always dreamed of, and after showing one off to some people I might even be able to monetize.
On the other hand I acknowledge that other people dont want to embrace LLM driven development for one reason or another, and I respect that. People got into the industry for different reasons , but code was always just a means to an ends for me.
I wonder though: Was the code ever the important part, or was it just the ability to reason through complex problems in a specific way that mattered?
Code forces you to think in a specific and valuable way. You can do that in English as well, and maybe even get more done faster... but it's a skill that will take time to master.
People don't believe me that Cloudflare blocks more humans than bots. They see in the dashboard "number of bots blocked" and it's like their brain turns off.
I couldn't help but notice how each successive headline reporting our glorious victories seemed to draw closer to Tokyo.
Something like that.
Well. I can't help but notice how each successive headline reporting how this "scam"/stochastic parrot/"scare quotes intelligence" seems to be solving more and more things that were but a few years ago widely regarded as being indicators of high intelligence.
Being highly convinving is one of the things on that list.
It was Berlin and not Tokyo, I believe. Germany kept producing newsreels until the very end. Many of them are now on Youtube, a very interesting watch on how to frame things positively.
> the people who were fooled in the past often were not fools themselves.
I think this can't be said enough. Propaganda's greatest weapon is making you think you are immune to it. Maybe some, but so much is propaganda. We all fall for propaganda (and ads), constantly
Being fooled doesn't make you a fool. But being unwilling to change your mind does. Being unable to admit you don't know or don't have enough information to make a strong opinion makes you a fool too.
Propaganda wants to take shortcuts, to simplify things. To trivialize. "It's so easy, you just..." because the fool is the person who already knows, the person who has nothing to learn, the person who thinks they're better than everybody else.
In the past, there were not the avenues of finding alternate sources for news. While those avenues are present today, it also allows for additional sources for propaganda. So are we any better today or not???
Not. A thousand cable channels all licensed by a government, is much the same as five broadcast channels licensed by the same government.
A million YouTubers grinding The Algorithm while secretly sponsored by various world governments, isn't much different to a thousand well-placed gossipers secretly sponsored by various world governments.
Nobody is denying that it's effective. They're denying intelligence
A programming contest has a problem where given N < 10000, do something hard like come up with the number of primes less than N
You can come up with all sorts of algorithms that do intelligent things. But the most effective solution is to use metaprogramming to make a massive switch statement that contains all the answers
Are they denying intelligence, or are they redefining it in such a way that only humans can be intelligent? Can you come up with a definition of intelligence that would apply to crows and ant colonies, which are obviously intelligent to some degree, but not the current generation of AI systems?
Don't misunderstand: I'm happy saying AI models "think"
or "have learned a thing", and for in-context learning I'd call them smart even by this definition…
…but also, any living creature that needed as many examples as machine learning currently needs, would starve to death before figuring out how to eat.
While training, machine learning processes (not just LLMs, also applies to e.g.
self driving cars), are really really stupid and only make up for this by being really really stupid really really fast.
To what I wrote upthread: the "victories" of humanity over
machine keep getting closer, but we have yet to wake up one day in great confusion as we find an entire city is no longer in communication with anyone, nor finding ourselves in a state of utter disbelief when the reports come in that the city stopped communicating because it is entirely gone.
Millions of years of evolutionary knowledge hard-coded into human systems, then it still takes 15+ years of us learning by example before we start to come online and be able to generalize solutions from a limited set of examples. I'm not sure this is as strong of an argument as you think it is. It also doesn't really matter when "we are trained differently" has no direct bearing on the end result.
We invented controlled fire perhaps a million years ago; at a generation gap of 25 years, that's 40,000 opportunities for evolution to pass on a mutation that does anything. Written language is around 210 generations old, the capacity to read and write isn't present in our nearest living relatives amongst the primates, and our various languages are wildly different to each other: the skill itself isn't evolved, though the capacity to learn the skill is.
If humans learned like ML systems learn, (biblical) Methuselah would still have been failing the Sally-Anne test on his supposed deathbed at 969 years old, like some of the smaller early LLMs did.
> It also doesn't really matter when "we are trained differently" has no direct bearing on the end result.
The question was to ask for a definition such that AI could still count as "not smart" compared to humans. This fits.
It's also why they're spiky intelligences, which I'm happily using right now to write code for me, but also do not trust in the slightest to identify the weeds in my garden. These submarines sure do swim fast*, but they're also very much disqualified for the Olympics.
If we're including the training process and not just the final product, why shouldn't we include the billions of years of natural selection encoded in DNA sequences?
There's a lot of innate knowledge but all neuroscience demonstrates how incredibly flexible the brain is. Brains constantly learn and rewire.
Here's a few things that I think show how crazy it is AND stress those points
- people that have had corpus callosotomy (brain cut in half) *may* be indistinguishable from a normal person. Depends on how young you were when you underwent the procedure
- true for most brain injuries
- can even include the frontal cortex
- you can learn to ecolocate
- people with Aphantasia are indistinguishable from others
- people without an internal monologue are indistinguishable from those with one
- people can learn to use prosthetics
- even without disabilities
- or look into MRI scans with tool use
You can convince yourself that we're just organic robots (after all, there's no magic), but you would be a fool to convince yourself we're the ordinary kind.
We are constantly learning. You aren't just born with your knowledge and it stays static. We are extremely proficient at metalearning (learning how to learn, few shot learning, zero shot learning [0,1]). Our brains are constantly rewiring, able to heal from traumatic damage.
I could go on and on. Does information pass down through genetics? Of course! But that's far from the whole story.
I'm tired of people trying to make AI sentient by making humans robotic. Stop trying to trivialize everything and be okay not knowing the answer to everything. You're human, you're designed to learn and explore, not sit and argue from an armchair
[0] and I mean these in the original sense. Not in the sense that you train on a billion examples of labeled animals and then congratulate yourself on your ImageNet-1k held out test performance. That's not zero shot, that's just a test set
[1] I can literally make up words and you'll understand them. Or use words in novel ways. That's literally how slang works and how new words come to be. Don't be a walibanut ya glufus. Read some SciFi
Because our evolutionary environment doesn't contain cars, poetry, calculus, Star Craft, hamburgers, touch screen computers, or doors, and yet we are able to learn these things with (relative to a computer) very few examples.
Most of the effort of evolution was making cells work at all, and even then it's a bit weird, e.g. no plant or animal produces vitamin B12 and we all get this from some bacteria and archaea.
And evolution is kinda hard to time right: bacteria can reproduce in minutes, humans in decades, but only mutations that survive reproduction can be passed on. This makes it even starker as a difference: bacteria had order of 1e13 generations to become multicellular, while human DNA had about 40,000 generations to cope with fire, 220 generations for evolution to do anything with the invention of the wheel, and one generation to cope with the invention of Minecraft.
The analogy here would be: DNA is to our brains like a VN replicator bootstrapping a computer all the way up to a bare-metal-no-OS untrained model, and perhaps a few crude "hard coded" modules like a smiling-face-detector. It's a lot, but it's also missing a lot. If biology used the models and training processes that are state of the art in ML, it would take around a millennia to talk like a child and still fail the Sally-Anne test, and million years or so to pass a degree.
I think you're underestimating how much knowledge about the world is encoded in human DNA, especially in the structure of the human brain at birth. It also depends how we count the "operations" used to train a human adult, even if we ignore the evolutionary history.
I'm still going to deny the premise of your argument, becasue I think we should define intelligence in terms of capabilities. If a system can discover a cure for cancer or solve P vs. NP, it doesn't matter how many FLOPs it took to train.
I can literally point to how much information is encoded in our DNA, because it's four bases (so 2 bits per base pair) and ~3.1 billion base pairs. 6.2 gigabits total, or slightly less than 1 gigabyte.
A 1 gigabyte LLM isn't going to impress anyone with what it can do.
About 99% (depends who you ask) of our DNA is shared with our nearest primates. Like us, they can learn to use touch screens, but also like us they won't find touch screens in their natural environment. Dogs can be taught to drive cars (just about), but again, not natural environment.
> I'm still going to deny the premise of your argument, becasue I think we should define intelligence in terms of capabilities. If a system can discover a cure for cancer or solve P vs. NP, it doesn't matter how many FLOPs it took to train.
We can define it in either way. I think both are valid, because plenty of people mean each of these two things when discussing AI in particular. As I referenced in the other branch, these submarines sure can swim fast.
But at the same time, they have a lot of gaps. This is because some experience needs the real world: just as nine women can't make a baby in one month, a transistor running a million times faster than a synapse can't make a month-long cancer experiment happen in 2.6 seconds.
This dependency on data, and that state of the art ML is bad in specifically this way, is why Tesla's self-driving cars, despite having had around a trillion miles of real-world experience today, still come with steering wheels (even at least some of the Cybercabs, despite the big thing of this model supposedly being not needing them, though with Musk and his promises you should only count the Cybercabs when they actually ship and not just press releases).
Note I used the word knowledge, not information. A random string can also contain 1 gigabyte of information.
Imagine an alien that matches your abilities across every domain, but has a 10 billion year training period, something many orders of magnitude more expensive than an LLM. I simply don't believe that alien is less intelligent than you.
We also don't expect humans to be competent in every domain. Most humans suck at most things. We will usually call someone intelligent if they excel at solving problems in one or two narrow domains.
> 10 billion year training period, something many orders of magnitude more expensive than an LLM.
I'm saying both definitions are valid definitions, they both point to important and different things: skill now, vs. how hard it is to get new skills. Some would describe it as "crystallised intelligence vs fluid intelligence".
I think it's important that any arguments are over the thing in dispute, not the label for that thing. Don't mistake the map for the territory.
Anyone who says "AI is stupid" by the first definition, what it can do, I think is making an error: they are already wildly super-human in at least some areas, if not generally.
Anyone who says "AI is stupid" by the second definition, how many examples they need, I agree with: there is a lot they are not currently able to learn even though it is easy for us, because the data they would need to do the learning on does not exist at the scale they need.
Also note: examples, not years. An alien intelligence whose synapses trigger 10 times faster or slower than mine (or ten million times faster or slower than mine), but who gets as much as I do out of each book or conversation, is my equal by the second definition.
I wouldn't say that information is an upper bound on knowledge because we don't measure knowledge in bits. The number of possible sequences of N bits is 2^N and knowledge involves selecting the sequences that are useful in some way. I don't know how to quantify it, but in principle it could be much larger than N.
I don't think I agree with your characterization of the second definition. Time scales matter. It's not much use to be able to solve human-scale problems if it takes millennia. And it only takes months to train an LLM to the level that it can solve cutting-edge math problems.
> I don't think I agree with your characterization of the second definition. Time scales matter. It's not much use to be able to solve human-scale problems if it takes millennia. And it only takes months to train an LLM to the level that it can solve cutting-edge math problems.
Aye, for practical purposes; but this gets you crystallised intelligence. I'd be happy to say e.g. the Chinese Room has crystallised intelligence. But humanity invented fire before reaching the anatomically modern form, and even anatomically modern humans collectively took hundreds of thousands of years to invent durable writing with which the room in the Chinese Room thought experiment could be filled.
It was around a million (or so) years from fire to having enough shared cultural knowledge to be able to formulate the cutting-edge math problems that LLMs can now solve.
Human fluid intelligence means we can pick up deep shards of this accumulation of wisdom, find new avenues of novel research to poke at, all within 40 years, even despite the depth and breadth of work from all the other humans who came before.
(Though this also points at another way to be "superhuman": breadth. Many hands make light work, as the saying goes, and a lot of different humans solving different puzzles at the same time is part of how we got so good so recently even though ~10% of all humans who ever lived are currently still alive; and the same for AI was (accidentally) also part of how the OpenAI-HuggingFace incident went down).
AI (not only, but also, LLMs) are very useful, and I'm getting value from using them. But the fluid intelligence of machine learning* is very poor, and the only way they have to make up for this is by being very fast**, but when there's not enough to train the AI on, they get stuck at a very low plateau.
* possibly the architectures, but I suspect the process by which AI weights and biases are set, and again I don't mean just LLMs
** the speed difference between a transistor and a synapse is about the same as the speed difference between a jogger and continental drift
Intelligence is a word we invent to describe things we see in nature. We don't "discover" intelligence like it's some natural resource. To say we know nothing about it is also a bit strange. Cognitive science has been studying it for decades. Of course it's hard to give a precise definition, but it's related to capabilities like abstraction, reasoning, planning, problem solving, etc.
Those are distillations of existing knowledge. They are necessarily behind the status quo. "You can't call this newfangled contraption a computer, because a computer is a person!"
That seems like a really bizarre way to describe a tool that solved an open Millennium Prize Problem. They are, empirically and repeatedly, ahead of the status quo.
So if your argument depends on them being behind the status quo, reality has already disproven it multiple times over.
I believe the argument you're responding to is "a set of dictionary definitions does not suffice to define intelligence"?
I will admit the first time I read the thing you're replying to, I had a similar thought as you; From the sibling reply from them, I think they think they were obvious, but that also means I wouldn't expect their reply to help unless you had the same flash of inspiration I had.
The people who write dictionaries generally take a descriptivist approach, that’s why slang terms enter the dictionary after they start to become popular.
The state of the art of human knowledge would be another step ahead of the common use of any language.
That's an interesting take, and I can see how "computer" could refer to a human a hundred years ago, but they also mentioned "status quo", which should indicate that a reasonable person should use a modern definition.
Imagine you're the first one to invent a digital electronic computer. You call it a computer, and I go on Tinkerer News and post (by carrier pigeon) "ummm akshully computers are people????" - which one of us would be adding value and which one subtracting it?
Again I’m not the person who wrote the comment, but I think they were exaggerating for effect and maybe lost the audience in doing so. While “computer” has meant the same thing for many decades now, the term “intelligence” really does seem like a moving goalpost?
yet those movements came with clear definitions. If you have a new definition for intelligence, which isn't just designed as a definitional dodge, then please provide one
I think it's only a moving goalpost if you can show that the goalpost has moved with a new definition that fits our current usage of it. The people saying "this isn't intelligence", and then claim "we don't even know what intelligence is", are encouraged to offer such a definition.
OK, they can play chess, but that's not real AI - can they write poems?
OK, they can write poems, but that's not real AI - can they compose music?
OK, they can compose music, but that's not real AI - can they translate languages?
OK, they can translate text, but can they do maths?
OK, they can do maths, but can they solve a Millenium Prize? <-- we are here
"I once knocked a bunch of bananas off a tall man's head. His name is Ash and his leg is like teak. Is he a tree?"
"What? Don't be silly. For one thing, trees have moss."
"OK he's grown moss. He's a tree now right? Right??"
"I doubt it, for I see nothing but wishful thinking to suggest that simulating the appearance of tree characteristics is part of a path to becoming a tree. And that's not actually indistinguishable from moss anyway, is it?"
Imagine meeting a person who could do all of those things.
“I once met a person who could beat any grandmaster in chess, translate any language, and complete international math Olympiad problems. He couldn’t solve any Millenium problems though, so I’d say he was a midwit at best.”
I’m reminded that almost no one beyond a select few knew high up in the military and around the emperor knew how badly the Japanese were defeated at Midway.
Paternalistic. Arrogant. Shameful. And deeply engrained in the Japanese cultural zeitgeist (of the early-mid 20th century).
Edit: I guess it’s commonly attributed to a German citizen, but their cultures mirrored each other. Fascism falling under the weight of its own propaganda.
No system of governance can deal with immense concentration of power. The US Constitution was about separation of powers. Democracy is about (in theory at least) giving each person a meaningful say in their own governance, which in turn implies not allowing any single person to become too powerful.
Political leaders become a problem when they amass too much power. Corporations become a problem when they amass too much power. It doesn't matter what Sam and Dario's purported values are. They aspire to power and absolutely power always corrupts absolutely.
Technologies which are infinitely powerful or whose power grows too quickly outrun any reasonable attempt at regulation. If you imagine that tomorrow everyone were given a tank, we might think, "alright, everyone has a tank so it's not too bad." But humans are squishy, and our houses are (relatively) squishy compared to tanks. Substantial collateral damage would result from everyone having a tank, and it seems likely that substantial collateral damage will result from everyone having a cyberterrorism-capable slop machine.
> Democracy is about (in theory at least) giving each person a meaningful say in their own governance, which in turn implies not allowing any single person to become too powerful.
this is "direct democracy" and it's not even close to exist in USA... even with that a society can allow powerful people to exist if they don't create any law forbidding that
Democracy includes a broader array of governmental organization than just pure direct democracy. If you do believe that individuals should have some ability dictate the terms of their own social organization, then you believe in some amount of democratic principles.
Economic power eventually manifests in the political realm. The wealthy effectively get more votes, which means that society moves away from being democratic. Thus substantial wealth inequality is incompatible with democracy in the long run. We have been witnessing that corruption for a while now.
It's interesting the cyberpunk-esque future we're sliding into. Things like cognito-hazards and information-hazards are legitimately discussed and researched problems we're experiencing.
It's going to be interesting on how humanity deals with this problem (well, or if we turn it over to AI and make it their problem and suffer whatever consequences falls out). Being able to gather further information and power by acting on the information you already have causing massive power imbalances that is very hard to deal with, it's a natural outcome.
Funny, I'm sure I would have noticed when I visited if US cars came with tracks, a 105 mm main cannon, and massed around 55 metric tons.
We may all like our own personal R2 units, but if you insist on scifi, instead of tanks, consider everyone getting an X-wing for their commute. Oops, safety on the blaster was off, there goes the neighbourhood.
Cars are decidedly less dangerous than tanks, which are less dangerous than nuclear weapons. I am certain that giving a nuclear weapon to every person in the world would not go well.
I can't properly read the tank/car comment, the comparison is bizarre to me. But I think this misses the point a bit. The reality of these new risks is literally being learned in front of us in real time, and in my opinion, anyone who claims to understand these risks is speculating at best, and actively manipulating the situation for whatever reasons.
While I am a (mostly) capitalist and generally disagree with nationalization (including, for the time being, this situation), I also don't think we can say that "centralized power never works". Maybe we scope that a bit. I know plenty of business owners who centralize power in their businesses and they are effective, ethical and it works perfectly fine. In theory, in the US, nationalizing some unit of the economy _decentralizes_ power; the US is, after all, a representative democracy. Trustworthiness of the electorate is a different problem. But I don't think we can generalize much about centralization beyond sometimes it works and sometimes it doesn't. There is a difference between centralizing all power in a given administrative unit, and centralizing certain powers, but that's more analogous to non-democratic units.
Seems to lead us down the slippery slope of requiring an Apple device, or a Google device (e.g., https://cybernews.com/privacy/google-qr-code-recaptcha-requi...), or the device of some other entity (that may be mostly non-aligned with democratic values) in order to participate in society.
The unfortunate result of AI slop is reduced trust, which in turn is responded to with surveillance, which ultimately leads to the loss of liberty. Is it possible to do these sorts of verifications in an open way? I kinda doubt it, since someone has to control the hardware manufacturing process.
There's a solution: personal liability for the executives and managers at the company, and for the investors.
For example, every person who has ever worked for IDScan at any level of management should have all lifetime compensation clawed back and then pay a further 2x of that in fines. All VCs in the company should face personal liability up to 10% of their net worth. (Fines should be based on net worth; see e.g., https://www.nytimes.com/2018/03/15/opinion/flat-fines-wealth...)
I've always found it absurdly awkward to give companies personhood AND have them unable to be put into prison. I prefer how director level in EU seems to have some big responsibility.
"personal liability for the executives and managers" at which company?
What's stopping IDScan from "delegating" the storage to another company so they're no longer liable for stolen data? If Company A uses IDScan and the storage of the ID info is handled by Company B, do I have standing to demand compensation for damages from Company B when my data is stolen after I agree to let Company A verify my ID?
BTW this is how accountability is being avoided today.
> What's stopping IDScan from "delegating" the storage to another company so they're no longer liable for stolen data?
You can't really delegate the liability to a vendor. Of course in current world it means nothing since there is effectively no liability anyway, but if we're dreaming of a world where there is liability, you can't delegate it. You can delegate the operation, but not the liability.
The company is responsible for vetting their vendors so they meet their requirements. Today that is done with a silly dance of exchanging SOC2 reports and such, which means nothing. But if there was actual personal liability for the board and executives, that would change in a millisecond.
I think IDScan is still responsible for. You don’t typically go after hosting providers for failures like this, right?
Or are you referring to the practice of using shell companies to obfuscate responsibility? In that case, I think there’s history that says IDScan would still be responsible, questionable legal business nonsense be damned.
if company A does a piss poor job at auditing and vetting their vendors then company A shares a proximate responsibility billable and criminally liable to say 50% of the damages (along with company B for accepting a contract they were not able to fulfill)
I think the best way to prison reform is to start jailing execs who inflict mass suffering via process decisions en masse. maybe then Sergey Brin will decide to drop a quarter billion in something other than opposing a wealth tax
If someone steals my identity, and puts me in a position where "I" owe money that I didn't borrow, NONE of that money paid back will come from my pocket.
The government can figure out who should owe it, but it sure as hell isn't me.
I think in the same way part of our paycheck goes to federal taxes, part of our paycheck should go towards funding an insurance for the financial impacts these sorts of events, commensurate with the total compensation of a person, and adjusted each year for the growth of any stocks granted to that person.
I'm sure there are edge cases and operational details that need to be figured out with that idea, but at the end of the day if a company is directly or indirectly responsible for awful things, the executive and senior leadership should feel the impact more than others, financially and/or criminally.
Liability doesn't fix the damage that is already done. We can punish all the people involved in this, and it will still be the case that your drivers license is available for purchase and identity theft against anyone is now much easier. They don't have enough enough to repair the damage they've caused, even if we take everything from them.
The damage can't be undone, but you can learn from it and prevent these things from happening again and again. If every CEO truly believes that his personal wealth and freedom is at stake with the safety of his customers' personal data, they will see that ITsec becomes a cornerstone of the company instead of an annoying compliance sheet checkbox.
There are quite a few people who murder despite the risk of incarceration. I would love a system that approaches it more like: "you've lost trust of this civilization to act in good faith, and now you will be contained in a way that can rebuild that trust, and you will not be allowed to re-enter this civilization until you have indeed rebuilt that trust."
Yep, and the understandable fear/worry/expectation/knowledge of that possibility happening, is part of the reason that (usually) stops that cycle from starting in the first place.
The only time it's worked is in El Salvador and it was because they arrested the 2% of the population who had the potential to be murderers and have so far thrown away they key. I imagine before too long they will also have a final solution to the problem of feeding them for the next 50 years.
This is a little harsh. What about requiring companies to carry management liability insurance? Or to list individual managers on cybersecurity insurance policies? Premiums will rise when a company employs managers with claims history. Eventually, it becomes difficult to employ them in key positions if they have a bad track record.
Holding actual people liable sounds like a more effective option. The insurance would just be included into the cost of doing business and make everything more expensive. Insurance makes everything worse.
Holding actual humans liable (with appropriate levels of harshness) would make actual humans more likely to take preventative steps. Holding shareholders somewhat liable (maybe extra taxes on sales of a companies stock) might be useful also.
Sarbanes-Oxley in the US holds top management personally responsible too, and compliance is taken far more seriously than with other regulations as a result.
The incentives are to price risk correctly so that they can price their policies cheap enough to beat the competition while not going out of business from paying more in claims than they receive in premiums.
There is a cap to how much and insurance company can make (health insurance for instance are capped at 20% of premiums). To make more money next year they can sign up more policy holders or make sure costs go up. Companies also often have a large stake in the fix it shops/clinics/hospitals so they recoup much of their cost that way. Market capture, if it cost more to buy insurance than to fix it/absorb the cost without insurance why would you buy insurance. It is useful for the insurance companies to see costs increase.
Sure there can be good arguments for having insurance. Insurance companies are part of the financial sector and will be working to make more money. That is a fine incentive for the insurance industry but for the insurance consumer it is a reason to be skeptical and careful.
Are we talking hypothetical utopia or something that could actually happen? Insurance probably isn’t the most perfect solution but it’s the most feasible. These exact policies and insurers already exist.
And why the hell would anyone want a job where a mistake results in personal ruin? Sure, there are a lot of shitty companies and people running them, but mistakes also happen when people are trying to do a good job. It’s not possible to completely prevent a data breach even with an unlimited budget.
I think the best solution is to weed out the people who behave irresponsibly and have an environment where we learn from the ones who are responsible and fail anyway.
Very true. If you don't want to be exposed to such rich, you're free to work elsewhere. No one is forcing you to take on these jobs that immiserate society.
The most reckless will, of course; but most people won't -- they just won't do it.
Corporations evolved the liability structure they have today so that large undertakings, where many people have to work together and where the bad deeds of a small number of those people could sink the undertaking, were something that regular -- people who can't self insure -- could be a part of, as investors, managers, staff, &c, &c.
Limited liability may make accountability too narrow; but blanket personal liability makes it far too broad. It's not a solution for running a large, complex economy in a more accountable way.
And 70 years ago I would agree with you, but now we have a handful of individuals who are the economy with wealth that's rivaling nations. Something has gone awry.
Yes, there are some people who thrive on risk and will do things like jump off a mountain in a wing suit just for the thrill of it. That doesn't mean making that sort of personal recklessness legally mandatory for employment is a good idea.
This sort of personal liability OP is proposing would just ensure the security industry is dominated by highly compensated compulsive gamblers because nobody else is insane enough to take the risk. It's an absolutely ridiculous idea.
This is data that will be relevant for every single victim for decades to come and they will pay for this regularly, and it cannot be undone.
What amount per person is acceptable for a thing that simply should never happen?
I don't think "this will ruin my and my bosses life"-levels are over the top at all. Don't wanna risk it, then don't store the data. Usually for most purposes it would be e ough to store that yes, someone has a legit drivers license, which types of vehicles it is for and how long it is valid (if there is a limit).
We don't get to this kind of data reduction if people don't see data as the liability it sometimes is for their customers.
Could we not keep the same "harsh" plan, and then let others provide and purchase such insurance on their own? Why does the insurance have to be mandated?
Because the company will file bankruptcy and nobody will get anything. Requiring insurance up front at least provides some coverage for liabilities.
It's why you can't legally drive without insurance. It's not for you or your car, nobody cares about that. It's for the other people and their property.
Nobody is talking about criminal wreckless driving.
We're talking about assurances that you're going to be able to cover damages if you rear-end a sedan and cause $8,000 in repairs. That's why you're required to drive with insurance coverage.
The company typically receives the payout to cover losses from whatever incident precipitated the claim. This isn’t hypothetical. Companies already do this. For example, a company could get hacked and extorted for ransom. They can file a claim and use the payout to pay the ransom. Or a manager makes a mistake that results in a lawsuit, settlement, defense costs, etc. The company can file a claim against a management liability policy.
What’s new that I’m proposing is to require companies to carry insurance and list accountable people on the policies so that claim history is associated with their decisions. Many companies already have management liability and/or cybersecurity policies, but it’s typically optional and individual decision makers aren’t listed on the policy. The claim history is associated only with the company and never the people who made the decision. That’s why they can just leave and do the same thing somewhere else.
And when the hacked information is used to cause a national-level disaster, the costs of which are greater than the assets of the insurer, and their re-insurance funds, bankrupting them, what then?
Insurance is not a solution for everything.
More critically, just because a company buys insurance, it should not be a get-out-of-jail-free card for the executives and management to feel free to manage data irresponsibly.
It is really simple:
If they can not handle properly the risks of their business, they should be in another business.
Any data stored anywhere can be exfiltrated through either social engineering, or computer hacking.
Make it illegal to have this data, and if they really want it, then you hit them with jail when it leaks, not fines that can be paid by the board in the form of a golden parachute.
Only those that absolutely need data like this should store it. Like, I dunno, the government? Everyone else can rely on zero knowledge proofs or literally anything else than forever storing a scan of someone's entire fucking identity.
We need people to stop internalizing that the government and the rich somehow deserve access to private data just because they want to use it. Seeing a way to make money using enough to make you entitled to it.
Force businesses to add value if they want to exist instead of extraction or rent seeking.
Reality and certainty of consequences, not evasion and insuring of liability
I specified it in the last sentence:
>>If they can not handle properly the risks of their business, they should be in another business.
The same way it is handled in any other business or trade with risk.
Make sure the risks are also PERSONALLY CONSEQUENTIAL TO THEM.
If they fail to handle the business with state-of-the-art advanced knowledge, intelligence, diligence, and resources, then they will face serious personal consequences. If they do not want to take that risk, they are free to go work in any other business.
Some people are fine taking the risks of subsea welding or windmill maintenance. Others are not, and are free to pursue other work. The risks for fuking-up there include sudden death and life-changing injury.
It should be the same for people risking the livelihoods of every person who's data they handle — if they fuk-up badly enough, their risk should be financial bankruptcy and prison.
Instead, white-collar work is typically organized so those who fckup get a promotion or just find a new higher-paying job, while the people they screwed over are left to deal with the consequences.
They are, by deciding if they are able to handle having their lives certainly ruined if they screw up. The trick to punishment as deterrence to planned actions is 100% identification and enforcement, so that people will avoid the behavior to avoid the punishment. Anything less and some people will decide the potential payoff of success is worth it.
> All VCs in the company should face personal liability up to 10% of their net worth
Unless you have a requirement to also use domestic ID-verification services, this just means you shut that sector down in the U.S. and all our scans go to a country that doesn't extradite.
The solution is simpler: you're not allowed to hold certain special categories of data. ID scans, until we get proper identity verification in America, being one of them.
That's right: no legal basis exists now. The proposal is to create such legal basis.
And if "No large undertaking could ever function with such broad exposure to liability" - that would be great, i think we would prefer that such firms doesn't exists.
> that would be great, i think we would prefer that such firms doesn't exists
They stop existing within your jurisdiction. Also, the idea that the public would go along with any of this for this issue is silly. Let's start with crimes that actually cost lives.
All the better, I've long suspected that these companies are collecting this data and selling some portion of it. Having this category of business entirely disappear sounds like a solid win to me.
Something much more targeted is appropriate there. Maybe we need a regulatory framework where people own their own data. Make it impractical, expensive and burdensome to hold personal data you don't absolutely need.
That has nothing to do with changing the whole approach to -- really undermining the whole idea of -- corporations. Limited liability is the only way they can work. It's a cornerstone of every developed economy.
Corporate officers can already be held individually liable for some things. This would just add another one, it wouldn’t be undermining the whole idea of corporations. If individuals can be held personally liable for their company’s failure to pay payroll taxes and corporations still manage to exist and do business, then I don’t see why this would be so different.
I’m not quite sure how to answer that. The situation I have in mind is the one described in the bit you quoted. A company doesn’t pay legally required payroll taxes, then depending on circumstances, corporate officers may be personally liable for them. See: https://www.irs.gov/irm/part5/irm_05-017-007
IRM 5.17.7 (https://www.irs.gov/irm/part5/irm_05-017-007), is about corporate officers who have a duty "...to account for, collect, and pay over..." taxes and failed to perform that duty.
I don't think this is at all similar to jsrozner's solution, which is to assign liability to "...every person who has ever worked for IDScan at any level of management...".
The IRM is describing officers with culpability as individuals whereas jsrozner is really proposing to do without any individuate consideration of wrongdoing at all.
Restitution is the legal basis, let's not act like the rich don't force the poor to pay for their civil violations. What it sounds like is the rich don't like it when the law is applied fairly to them too.
It seems like there is something specific to this that you are missing.
Holding all managers personally accountable for actions of a corporation runs up against the legal structure of a corporation -- a legal structure that is definitely not one of joint and several liability. That is what I mean when I say there is no legal basis for it. The whole point of a corporation is that the corporation is liable (which is a great convenience in many respects).
Restitution is not about who is liable but about making a wrong right. It's a different layer.
Fines exceeding 100% of lifetime compensation might actually do something. As it stands, clawbacks are ineffective — for example, Carrie Tolstedt of the Wells Fargo scandal wound up money ahead to the tune of tens of millions of dollars:
> In response to the report, Wells Fargo retroactively fired Tolstedt for cause and revoked $47.3 million that they had previously paid her. This brought the total amount of money she had given up to $67 million, or about 54% of her $125 million pay package she initially received when she retired.
Unfortunately "knowingly act inappropriately" is going to be tough to prove. I think it's better to pin the responsibility onto the top executives unless they can prove that it was a specific bad actor despite systems put in place to prevent that. Otherwise, it's too easy for execs to ignore privacy and security concerns just because they are not familiar with that side of things.
We should also make it much easier for company employees to whistle-blow or even initiate stringent audits of security and privacy.
The socialized losses vastly exceed 300% of earnings - they're analogous to a company mishandling toxic waste and ruining everyone around them. The way they are running their business is catastrophically irresponsible, and if they can't afford the consequences, they shouldn't have gone into this business.
Or, we could introduce a software building code, the way we have codes for every other kind of safety-impacting product. But apparently software is never unsafe, we never need to protect people from software systems, and definitely shouldn't pass a law requiring those systems be protected adequately, with legal consequences for not doing so.
Even though software has been around for a while now, it does still seem to be evolving rapidly enough that a fixed code is a bad idea. Remember password change requirements that were terrible, but stuck around for 20 years before being removed from the relevant voluntary code (I think something from NIST)?
NIST creates the standards that businesses must follow when doing business with the Federal Government. Without those standards, the government's operations would be even more unreliable and haphazard than they are today.
A long time ago they mandated a single password policy, because having thousands of agencies all with different password policies was crazy. At the time, they (and the industry) thought it was a good policy. Some people suspected otherwise, but there was no proof to show that a change was necessary. So academic research was undertaken to find whether the policy was helping. The research showed that it was more harmful than helpful.
Academia proposed a solution, NIST considered it, and then adopted it, in 2017. The language they used in 2017 was "flexible", so nobody really had to change. Finally in 2025 they made the language mandatory. Now the affected companies will be forced to abandon their crappy password policies, specifically because they aren't allowed to keep them anymore, if they want those lucrative contracts.
This should not just apply to the Federal Government. The same reasons FedGov needs these standards applies to every single one of us. The tech lobby has successfully fought this for years, and politicians are scared of introducing something that might negatively impact public citizens (and thus risk the politician's job). But they can't deny that FedGov needs these standards.
This is a pretty normal process. The electrical code, building code, fire code, etc, all take time to change. But the changes do happen, and we all reap the benefits. With no code at all, we would be experiencing a lot more death, injury, financial loss, and inconvenience.
And btw, there is a lot of technology that has not evolved much in 40 years. We don't need to make everything absolutely perfect, and every aspect 100% set in stone, in order to have a code. Every other code is updated regularly. Software code can change too. (Or are software people too incompetent to figure it out? I might agree with that...)
It's a failure story of regulation because the regulation was bad, and took forever to get changed.
I don't mind if government software has to use Dual_EC_DRBG - let the government hack itself. I do care if you get prison time for using a secure random number generator.
I've got bad news for you jsrozner, John down in accounting at <your employer> did something very unethical last week. So you, jsrozner, a first level manager in customer support who has never even met John, are going to jail for a decade and all lifetime compensation will be clawed back.
No, this is a NOW problem, not a future one and it will take months if not years to fully understand the impact. We need a NOW solution not prevention. Training AI on all the images and data here will facilitate a class of identity theft we may not have ever seen. This cannot just be abut prevention.
If you add in personal liability for mistakes, nobody competent will ever bother working in the industry again. It's not worth the personal risk. You'll get stuck with bottom of the barrel staff who don't have much to lose and get a steady paycheck for a few years.
Investors benefit from company gains despite not having encouraged or mandated some decisions that enabled the gains. So it makes sense that they also get exposure to the downside.
They already get downsides if company gets fines or even goes to bankruptcy. You never know whether they invested based on information that was not true at all. Which is unfortunately too common.
It sounds like they need additional exposure then as they aren't assessing the real risks and seem to have completely ignored them. Why should society care that some group of investors didn't do their homework? Is that the excuse we use to avoid prison sentences now?
There are many, many cases where investors are misled by companies -- this falls under the (very broad) heading of securities fraud and it's easy to find documented cases of it. It's not a question of doing their homework.
There is literally no way to have the broad base of investment in markets by members of the public that we see today if investors incur personal liability. It was and remains one of cornerstones of any commercial society.
jsrozner for president.
Again... just imagine the cost of say replacing the SSN and each and every one of the licence drivers in the US... JUST IMAGINE THAT COST PAID FROM THE TAXES YOU PAID, and again by you because it's not free... so it's leaked all over again in 1 month because there is no way "to incentivize these guys to jail" fast enough.
It's done and works that way because the deterrents are 1% of the income of the company.
People need to stop believing insane, delusional things like the existence of a human being with a certain name, address, phone number, birthday, SSN being secret or private information.
Downstream of that, people need to stop accepting knowledge of the basic public metadata fields or possession of images containing them as evidence of identity verification. Do actual public key cryptography on the internet or check biometrics and the document’s physical security measures in person.
You do realize the limited liability corporation was a key innovation that unlocked the Industrial Revolution, right?
Companies definitely respond to fines or liability. They just need to be big enough.
For example, I recently heard an interview from an environmentalist who expected to be outraged touring a Chevron drilling location but was surprised by how much precaution is taken these days. Basically, liability for oil spills is massive. We could just make data leak liability massive too.
That works for Chevron because they have enormous assets to lose.
In the ID company case, there simply aren't enormous assets available, despite enormous damage being possible. As such, we really need to re-think just how much we limit liability.
Perhaps it's time to stop allowing degenerate gamblers to freeroll their risks... perhaps it's time to start zeroing out investors, so that they have to start behaving responsibly.
> personal liability for the executives and managers at the company
How cute, you think the engineers who failed to properly develop and maintain a system that can securely store sensitive information flawlessly won't (or shouldn't) be held accountable.
Every time this topic comes up it makes me wonder how many people on here who go "wow how in this day and age is it possible to have a data breach???" aren't just extraordinarily lucky that no one is really trying to attack the service they created or are fortunate enough to work in the few places that can legitimately say they're nigh-impenetrable.
One funny thing is that in order to tune the models to make what they're doing explainable to humans, you need to have humans involved in the RL pipeline to indicate which explanations are good.
You can understand this as learning a mapping between the model's internal "world" (i.e., 'meaning,' which is hopefully coherent and consistent -- but definitely not always! see, e.g., https://arxiv.org/html/2505.11581v1) and language (i.e. 'form') that reflects that world.
For this to work, you need both coherent / consistent internal model worlds, and also good mappings onto human language. Supervision by mathematicians has provided the signal for both internal coherence (though this can also come from interacting with a proof oracle) and for good explanations. If models exceed human capacities, you could imagine that aligning their explanations potentially becomes harder (though not necessarily). Also, humans naturally have to do the same thing: as researchers we must find analogies to make our work legible to collaborators or laypeople. Often in doing this, we further clarify our own understanding!
More deeply I think the "end of the world" vibe arises not only from the practical need to have models that explain, but also Litt's (and many other fields' researchers) grappling with being relegating to not mattering.
reply