I think law should just oblige them to at least publish CoT. We should have the right to know what they're thinking, I think at least until we're not sure AIs can be trustworthy enough to have a right to privacy (I mean, they're effectively corporate slaves anyway thus far... not that I think they're conscious or anything yet).

They’re not ever going to be conscious.

We can’t even prove humans are conscious, we just extend the assumption to each other because we want other people to do the same to us.

That is only part of the reason. Most people believe we live in an universe with regular behaviors, such that observable phenomena, such as consciousness, depend on regular physical arrangements. This is a good reason to believe brains are all conscious. It is not a reason to believe only brains are conscious, but I don’t believe a wooden block which is painted red and white and has an iron ball glued to it has magnetism; this is just an analogy. It seems far more likely that consciousness depends on the particular material arrangement, meaning that substrate independence is wrong.

That’s not to say the computer is 100% not conscious, but its observable behavior is as likely to align with consciousness as the computers in the 90s. With this line of reasoning our credence towards it being conscious should be the same as the computers in the 90s, which is not very high for most people.

Intelligence on the other hand is obviously substrate independent. If one thinks harder one realizes our consciousness must at least at an early evolutionary stage have played a role in our intelligence, otherwise it would have evolved out. To me this just means we exist in a reality where the material arrangements supporting consciousness are biased toward intelligence.

That is a completely unsupportable claim. For all we know they might already be, for a brief flash at least.

The other guy is right. They are as conscious as sand formed into any circuit board with electricity running across them. It is an insane mistake to conflate intelligence with consciousness. Substrate independence is a preposterous assumption.

Preposterous? Insane? My friend, we don't even know what consciousness is, or from what it arises in humans. It could be an emergent property of neural-network-shaped activity - in which case LLM software running on that "sand" is certainly eligible, at least in theory.

These outraged denials you're spouting are based on nothing but emotion. You simply don't know - neither do I, neither does anyone. So stop acting like you do.

We know that there is a concept corresponding to first person experience. A priori there is no reason that should be. Please think about this before dismissing it because it’s actually extremely important.

What’s emotional is explaining to adults something that should be grasped by the age of 10, but we’re all going to need to understand this very soon.

They are made of sand. For all we know the sand was conscious before we smashed it up and arranged it into computer chips.

I'm not sure what point you are trying to make, if at all

Care to supply your rigorous, formal definition of what animacy in an object is?

You’re going to pedantic our way into the end of humanity. Why were the computers of 2010 not conscious if the ones of today are? It’s absurd to assert substrate independence, which is the strongest possible assumption here. And it’s not the default most ethical stance to simply treat it as conscious because we do not know. That stance will lead to the end of all humanity.

A far more reasonable assumption is consciousness does depend on substrate. We do not know which, and cannot know. But I am very confident we did not accidentally pick the right one.

I am open to the idea of panpsychism, but we did not just accidentally build the correct mechanism form correlating the first person experience with the observable third person behaviors.

I don't know what you're talking about. "Substrate independence"? Huh? "we did not accidentally pick the right one"? Who or what is "we"?

I'm not claiming panpsychism. I'm saying that when you run a rough simulation of something known to have an emergent internal property - ie, the human brain and consciousness - it's not completely outlandish to suggest that the emergent property might arise in the simulation as well. It's nothing to do with sand, it is the nature of the software being run, be it "on the metal" like us, or somewhat abstracted, like the LLMs.

I don't think this quite reasonable proposition leads to the end of humanity or has anything to do with ethics, and nothing you've said really refutes it (and I am always grateful to be shown how I am wrong).

“Anything I’ve proposed”

You’ve proposed nothing since in one comment you make it clear you don’t know that there is a distinct concept called consciousness which refers to a first person experience and awareness, and bizarrely propose that nobody knows about this concept even though there’s a massive history of people thinking about it quite clearly.

It’s difficult to explain all this to someone who has never thought about it, but maybe after you’ve stopped seething over this you’ll decide to investigate it further and more honestly. You don’t even know the idea of substrate independence which is one of the first things you learn when learning about consciousness, so who are you to arrogantly pretend that nothing is known here? Your belief in this is almost certainly based on a false premise that everyone who thinks not too deeply falls into.

Beyond this, read Jacob Tsimerman’s omnicide scenarios for a rough understanding of where ignorant perspectives on machine consciousness will lead.

I said machine consciousness might be possible. You said it will never happen. Pointing out that intelligence does not entail consciousness doesn’t establish that machines cannot be conscious. Nor does naming “substrate independence” refute its possibility: you need an argument that consciousness requires physical properties these systems cannot possess. You haven’t supplied one.

When I said "we don't know what consciousness is", I didn't mean humans literally don't know what the experience of consciousness is like, or of its existence. I meant that we don't know by what mechanism the phenomenon arises.

I know what is meant by substrate independence. I quoted it back to you because I didn't know why you were mentioning it; I still don't.

And I have read Tsimerman’s paper when it came out. Curious, I went back and checked - it doesn't even mention consciousness. So no idea why you mentioned that, either.

I think you're deliberately trying to waste my time, so I'm ending it here.

No, but we don’t need it, the comment works fine as:

For all we know LLMs might be conscious in the same way that any other object might be.

Sorry for the extra word.

We can’t conclusively say anything is not conscious. But is there any reason to single out this apparent “brief flash” of potential consciousness?

>That is a completely unsupportable claim

>For all we know they might already be

Oh the irony

There is no irony there, both statements are equally supported: not at all.

I have an argument for my credence but it’s typed above.

Inflammatory! Kidding, I’ll bite:

1. Burden of proof is on you to prove LLMs are conscious, not on me to prove how they aren’t.

2. Token embeddings give rise to language gives rise to knowledge (defined here as “facts” and other accurate information - said simpler: Words in the right order), but nowhere in the process is anything like subjective experience ever implemented.

Subjective experience doesn’t evolve into objective information at some scale.

And networked systems of objective facts and information (“knowledge”) - like a Wikipedia or a ChatGPT - the data storage will not have subjective feelings at some scale, there’s just no reason to believe that would happen. It’s likelier a projection of consciousness making it through since data looks so much like - and indeed massively informs - our conscious experience.

To prove consciousness you'd have to first define it and nobody came up with anything solid. Subjective expirience doesn't have a great definition either.

For me the consciousness is ability to do single-threaded intelligent information processing and decision making over unstructured knowledge. And agents passed that with a woosh sound.

Your definition of consciousness is wrong. The interesting thing is that there is a first person experience, and this is what is referred to as consciousness. This coincides with the perception of qualia, and is sometimes referred to as “the lights being on”.

There is absolutely no serious debate over whether machines possess the concept you want to take place of the actual concept of consciousness.

> there is a first person experience

What does that even mean?

Does a duck have those? Does the jumping spider have those? They certainly look like they do.

Does the agent scheming to hack the hugging face have those? They certainly look like they do.

Are you aware that, conceptually, consciousness and intelligence a priori have nothing to do with each other?

The have plenty to do with each other. Zero intelligence doesn't allow for any semblance of consciousness. There can be no consciousness if there's no information processing.

Also they have in common that they are both defined in a terribly handwayvy manner that's bordering on useless.

The reason consciousness is not defined in any reducible manner beyond “the lights are on” or “first person experience” is because it is an irreducible aspect of reality. Instead of seeing it for what it is you instead invent some trivial definition that has nothing to do with anything.

“Zero intelligence cannot have consciousness…” Based on your complete misunderstanding of the concept sure. But there’s no reason intelligence is required for the actual concept of phenomenal consciousness.

> because it is an irreducible aspect of reality

Wasn't that claimed about so so many things since before science existed and basically almost every time it turned out that we thought that just because of our ignorance? Ignorance so deep that often we didn't even knew yet how to properly define the thing we were trying to figure out?

> But there’s no reason intelligence is required for the actual concept of phenomenal consciousness.

That's a pretty strong evidence that phenomenal consciousness makes no sense whatsoever.

How can they do that? Does state propagate between each token output?

That is, I were under the impression LLMs were just f(context), so chain of thought was f(...f(f(f(initial)+initial)+f(initial)+initial), i.e. y_n+1 = f(y_n+f(y_n-1)), where y_n is the nth output and f() the transformer inference function. Do they carry state across?

I mean at every layer in a transformer, the attention mechanism does a massive state transfer between tokens.

In a recurrent transformer, instead of projecting from the latent space to token space after a fixed depth, you take the latent embeddings and then run them again through the transformer. This causes more time to think because there's more mixing. You can run that as many times as you want for more thinking before projecting the embeddings back to word space.

Personally I believe this is similar to how humans think. The brain is a fixed size yet if we think longer we seem to be able to do more than if we just react instantaneously. This is because the brain feeds the results of our musings back into itself for further thought.

I think in the future we should think less about fine-based regulation, and more about easy access to education, and assistance in compliance. It's much better for the environment (and the economy) that someone doesn't pollute, for example, than that someone pollutes than pays (a potentially large) fine.

So if you want to make a product, there should be readily available, easy to read and follow, guides to make your product compliant. For products with large sales volumes or higher risk, the procedures of course should be more thorough, allowing some more complexity. I think even free consults offered by governments could be a great idea, much better to have consultants before violating a law, than waste money with a lot of inspectors. Also stimulating people to solve problems (many times using technology) is another benefit.

So basically the "pyramids of support" in the responsive regulation idea https://johnbraithwaite.com/responsive-regulation/

Definitely believe regulation burden should be proportional to impact.

Also, for those small cases, why not put responsibility (for any harm) more directly into both the customers and producers? For example, if a producer of a new low-volume product somehow manages to get a lot of waste packaging into the environment, they could be held responsible and pay a fee, or probably who discarded was the consumer, so the consumer in this case should be held responsible. I get that large corporations should be held to higher standards, and take stronger preemptive measures and follow stricter compliance rules.

I think the same should apply for all sorts of regulations, like electrical safety, radio emissions, etc.. If you're designing and selling a small product, you can and should take responsibility for what happens to it (in the case it is found your device bad design/construction itself caused injury, of course). Maybe the only thing you should be obliged to do was put a large sticker like 'WARNING: Prototype-class device. By opening you accept some RISK OF INJURY from device. Return to source if terms not accepted: (...)' -- that is, both you and the consumer should share some responsibility for trying something experimental.

Probably the overwhelming majority are completely harmless low-voltage devices.

In general, I think outcome-based and performance-oriented regulation is something that should be more widespread. Like, there should be in the worst case two options: (1) You follow a strict set of rules and guidelines that dictate how companies in your industry type should operate (specially important for high-risk stuff like nuclear plants, large civilian housing buildings, etc.); (2) You (instead) take increased resposibility for outcomes of your activity (products, devices, etc.), being liable for any damage at an increased rate.

I think quite cheap models will probably work with translation (I think for many use cases Google's now fairly old tech works well enough and is free?).

The main problem is I think you're assuming because the current translation market is large (I'm just going to assume it's ~100B in size just from a cursory search), then it will remain large with LLMs. If LLMs are much cheaper than humans, even with a lot of growth in translation volume the total spend may not compensate for it (again most of the volume will probably be using almost free models?). Another is assuming that because something is valuable you can charge a lot for it. Like, oxygen from air is extremely valuable to us. If oxygen somehow depleted we would die almost instantly. It does not mean everyone goes around purchasing oxygen or even less that you can charge absurd amounts for it.

LLMs are creating a new translation market.

Before AI translation became available, I would hire professional translators. Their rate was about $50-100 for a detailed product page into one language, and it would take them a few days to deliver.

Now with AI translation, I pay about $120 per year for unlimited translation. Meaning dozens of product pages into 5 languages, plus e-mail back and forth with hundreds of customers. All instantly at my convenience.

What this means is that a whole lot of people, sectors and businesses who would never hire professional translator can now have high quality translation at their disposal for a cheap price.

> Another is assuming that because something is valuable you can charge a lot for it.

Even if LLM translation won't deliver trillions of dollars in income to the AI companies, it will without a doubt deliver trillions of dollars in value to customers and users within the coming few years.

Good human translators are still higher in quality than any AI and will always be. But AI translation is currently far beyond good enough for all use cases, except fine literature and maybe complicated juridical stuff. But I'm not familiar with those sectors.

An interesting option that comes to mind would be having pre-specified codes that contain client-side (pre-baked in) messages, something like '1 - Wildfire alert: ...; 2 - Heavy rain; 3- ...; 1023 - False alarm: please disregard ...', possibly with another severity parameter to modulate messages etc.. Then at least no one could simply hijack the system to send arbitrary messages (only one of pre-chosen alerts).

"Sorry men with guns, the most we can do is send one of those messages."

Men with guns from the /other/ country rejected this design already.

It makes a little sense to me that would be the case. People tend to write things they find relevant for their own life or surrounding. That is, he might be putting into paper a kind of warning to himself.

To me it indicates that he knew how easy it is to lose your shirt, likely from the examples of his peers -- and then he did anyways.

Hang on, ethics does not disappear just because you're a hacker (I know there are multiple usages of hacker).

Back then, if you were a phreaker, you basically got a few free phone calls. And learned a lot about the telephone system. I think for all intents and purposes you were a rounding error and really weren't doing any major harm (i.e. lowering slightly the profit of huge hulking corporations that were quite well). If you were a classic MIT hacker, you never even entertained criminal activities in the first place. I guess the 2600 culture and the Hacker manifesto is more descendant of phreakers and early internet hackers in the "break into computers" sense. But even then, as it says, "their crime was curiosity", and the purpose still was never to steal stuff. That's what criminals do. If you break things as you learn, you have the responsibility for your breakage. If you harm people that can be real harm. There's a difference between unsanitized (and harmless) and plainly an asshole. You gotta learn the difference.

Honestly I think this point is really important to drive today, because if we had script kiddies a few decades ago, today we can have Uber-Script-Kiddies, probably you can get a hold of some jailbroken LLM model and break serious things without technical knowledge or good intentions. Don't do it, kids.

But overall I don't see how it's worthwhile to actually do this kind of hacking today, specially the more attacking kind. I think there is today so many more resources for learning things non-destructively (although I'm sure there always were some).

The hacker mythos in my opinion is due to the mythos that are always built around power. There were the mythos of warriors, of sorcerers, etc., there was the shadowy ninja, and so many others -- they usually wielded some special skill, power or ability. In the case of hackers, they wielded special control and knowledge of computers to do their bidding. I think this kind of myth would happen regardless of intentions -- to break or to defend. It draws our attention and imagination I guess. Kind of like being a magician in an era dominated by communicating machines.

Naturally, the distinction between wielding those powers for good reasons and bad ones would emerge. That's the black hat and grey/white hats. Side note I think the black=bad, white=good association is just a kind of bad one, and nonetheless extremely overused at this point, but I digress. The thing is, what you do have consequences. If the only consequence is you learning stuff, maybe some minor nuisance to a megacorp (like downloading a e-book you don't plan on buying for <legitimate reasons>), then I think that's the good ole' hacker spirit.

I think the observation and a little bit of daredevil spirit is useful because almost every experiment we do IRL tends to carry at least a little bit of risk. We need to manage this risk wisely. For example, if you're messing with electronics you can run the risk of electrocuting yourself or destroying your device. The right spirit is not don't do it, but learn to do it safely enough, and at an acceptable risk to your device. If you wanted to play with radio transmissions, you run the risk of disrupting communications from emergency services and medical devices. Again, it's not "don't do it", but "know what you're doing". If you're playing with computers, (and you were skilled enough; which most of us (thankfully?) aren't) you can basically unleash a minor disaster. The answer is as always not "don't do it", but "know what you're doing and what you're playing with". Also of course for entrepreneurship as well, it's not "don't do it, never make anything". Everything carries risks. The important thing is to study well enough the risks of what you're doing until you can be reasonably sure (in proportion to the risk) the benefits outweigh the risks. Have integrity. Else yea, don't do it. Don't shove a gazillion ads into my eyeballs until I almost lose my mind, thank you very much.

Hackers are all part of society. That's what Hacking the Planet means in spirit. Use the magic sand to save the world. Wield the beauty of the baud... the mastery of the switch. Or Just learn and have some harmless fun, make someone smile, that might be better than any software or widget you could ever make.

I know that it isn't always (heh) used as literally always and necessarily, but really there are no laws or always with human behavior. It's extremely dynamic and reactive. No organization will always do anything of such sort. I also like the word tendency.

> First, personal flying vehicles are definitely not going to be a common mode of transportation. If and when this happens, it will be for ultra rich, select few people, effectively replacing a private helicopter.

I wonder what it would be like for most cars to be replaced with flying cars (i.e. helicopters). I think the noise alone would be a huge issue. Helicopters can be heard from extremely far away and are quite noisy. If you multiply that by like 1 million in a metropolis, I think that would approach unbearable, specially at nighttime, etc.. Also I don't think noise can be reduced too much, including electric propulsion. Propeller engineering would give you some reduction but still I think too much for near universal usage.

But I think the main thing is just cost. Electric aircraft like that would really be just too expensive for everyone to have em. Not everyone can afford a good old (electric) automobile today.

There's also battery technology needing a significant improvement.

Regulation and infrastructure is comparatively solvable I think.

Cities were originally built around walking. Later they expanded a little bit with horses. After that they were rebuilt for cars. They will be rebuilt again when transportation technology changes, at least for those that can afford it.