Yes, exactly, that is my understanding as well. Since the watermarking is based on a symmetric key by design (as to not be easily able to test and remove the watermarks), the providers will need to offer a watermark detection API.
I see even more problems with this. To check any text for those watermarks, it needs to be sent to dozens of AI companies to check, potentially paying them all for just determining whether it matches their watermark, and more concerningly sending all that mostly human-written, often high quality text like unpublished research or books, to AI companies that almost all proved to obtain training data through all kinds of dubious ways.
My biggest concern is that checking any text for watermarks requires sending the entire text to Anthropic. And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, Mistral, ...
So every check requires sending the text to as many AI providers as offer a watermarking detection API, almost all of which have a very dubious track history with obtaining training data through illicit means.
Any university using AI detection in their submission pipeline, or lawyers, editorialists, proofreaders that check for AI marks will be sending significant amounts of text like unpublished research, books, potentially internal documents and more, most of which is high quality human written, to dozens of AI companies, blindly trusting they won't train on any of that.
Would really like to know how their watermarking technique works, and if they can use it to store arbitrary information in the text (I assume they can if only in a limited way). I assume if the text is long enough they could add all kinds of metadata that would then be undetectable as the model that generated the text is the "cryptographic" key that encodes the data. I wonder if you can have another model run over the text and destroy the watermark. I predict an interesting cat and mouse game to develop.
The paper for it is open. The technique isn't really hiding information in the text itself, but by forcing some of the rolls to follow a specific pattern. LLMs work by estimating the most likely next token, so there's sometimes a list of possible candidates that would all work in the text (e.g. synonyms). At low "temperature", the output is a bit more deterministic and otherwise it's a weighted dice roll of which token/word to pick. An LLM can loop over existing text and figure out if the output matches something it would do, similar to checking chess moves against the best computed move for detecting cheating. But the LLM purposefully creates a pattern of alternating weighted rolls that are highly unlikely to appear in normal text, and that becomes the watermarking.
The upside is that this has very low false positive detection rate, but the downsides are many. It only works on longer pieces of text. The system is fragile, and small edits (or rewrites by a local model) can fool the detection. Only the owner of the model is able to re-run inference at this level, so data must be sent to them for evaluation. And sometimes the token output is basically 100% deterministic because the input asks for the straight answer to a fact, or to recite a quote verbatim. That leaves no room for watermarking at all, unless the model is able to lie.
A way to think of it is that any time the model faces a choice, it leaks some fractional bits of information. Instead of making the choice randomly, you can put information in those bits.
But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty. Exotic prompts ("Use no more than five E's in four consecutive words anywhere in the text") seem like they will confuse the hell out of attempts to extract the bits from the output text alone.
> But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty.
When you only need to encode one bit, the signal to noise ratio can be very low. If I try to write my own human words under the policy of "try a little bit to avoid the letter 'e' in every fifth word," then a sufficiently long text would still be 'watermarked' even if I only succeed in this dictum (e.g.) 10% more often than the baseline.
most people have at least two, but generally around four to six (or more, thanks internet) interactional styles (not "selves", just things which vary depending on context, subject, and people they are interacting with). probably more. some of it is due to simple physical comfort levels (right now I am in a physical position where capitalizing is more difficult); some might be due to talking to a peer group instead of a group of kids or a priest or boss at a job, etc. unsure how that will shake out with AI but it unnerves me.
There's more to it than encoding one bit. You also want to avoid false positives. You can encode a single bit by XORing all the bits in the UTF-8 encoding. But then you get a lot of incorrect hits. The lower your tolerance for false positives, the more it acts like you're actually requiring more bits in your payload.
The scheme that Scott Aaronson describes essentially uses a specific prng, and you can then check a certain function with relatively few tokens to get a sense of whether or not a model using that scheme generated the text
A few caveats: you need to know the key to the function (used when generating the text) and you need to know the bias it would introduce
The point is that you do not need to know the full prefix, just a modest sample set of contiguous tokens
In fact such ‘arbitrary’ constraints uniformly improve composition. Thus eg if I force a - largely arbitrary - technical glossary to be unrelentingly applied to a translation, every single sentence improves in quality.
> The system is fragile, and small edits (or rewrites by a local model) can fool the detection.
Well no, small edits wouldn’t fool the detection as long as the seeding only uses a small run of previous tokens.
And yeah full rewrites breaking it is by design. The watermark is just meant to tell you whether the text was generated by a watermarked model, not whether the ideas came from AI or something like that.
In practice, it is theater. Are they going to do this with the code output too? This is nonsense security theater for the low thinkers to have a sense that someone is in charge. When we all know nobody is in charge, anywhere.
It knows when several tokens are about equal vs. times where one token is vastly preferred. In the latter case, that’s usually code or math or something similar and so it won’t alter those tokens.
I wonder if that is entirely true. It is also in the AI companies' own best interest to be able to detect AI generated text so that they can avoid training on it ("Habsburg AI").
In the case of Anthropic it would also be entirely unsurprising if they've been lobbying the government to force everyone to do something in their (Anthropic's) own best interest.
> It is also in the AI companies' own best interest to be able to detect AI generated text so that they can avoid training on it
It’s also in their interest to demonstrate that they can be trusted and to show that they at least pay lip service to limit the obvious downsides of the tools they are selling. The use cases they sell to mainstream audiences are not affected by detection tools. The point of having a LLM do the work for you is that the work is done, and reliably. It does not matter if it is done by a LLM, and most of the time it is obvious anyway.
The EU mandate for AI watermarking is likely the first step in the direction of prohibiting AI for specific use cases. The pretext for outlawing (or at the very least controlling) the use of AI when the time comes will be something along the lines of data integrity or just general compliance legalese.
So, the use cases that are being sold to mainstream audiences actually will be affected by detection tools, especially if the output is intended to be monetized in some way. In the near future the EU will likely come down with heavy intervention to prevent AI from impacting employment rates across Europe. The number of legitimate use cases for costly frontier models drops significantly once eliminating professional jobs is off the table. This is all conjecture at this point though.
I’m sorry, this is going to be a bit long but you made good points.
> The EU mandate for AI watermarking is likely the first step in the direction of prohibiting AI for specific use cases.
I am not sure how practical that would be. The cat’s already out of the bag and they won’t prevent companies in the whole world from releasing open weight models. Playing catch up by distilling flagship models is also relatively cheap; we’d see smaller companies setting up shop in friendly regimes. And I don’t see any appetite to go full child porn and criminalise the possession of a LLM. So we’d end up with a similar situation as with illegal downloads, i.e., everyone will do it and nobody will care.
> The pretext for outlawing (or at the very least controlling) the use of AI when the time comes will be something along the lines of data integrity or just general compliance legalese.
They could forbid using LLM for hacking, but hacking is already illegal. They could make it a factor when determining punishment, but I don’t think that would work terribly well. Most of the dangerous stuff we can do with LLMs is already illegal, or should become so. Things like propaganda, identity theft, harassment, scams. We need enforcement with teeth on these, not pointless feel-good legislation. Again, there are parallels with cryptocurrencies and torrenting software. These things have illegal uses, but it’s also really difficult to make them illegal, at least in semi-functioning democracies.
> So, the use cases that are being sold to mainstream audiences actually will be affected by detection tools, especially if the output is intended to be monetized in some way.
I don’t know that mainstream audiences are really against LLMs. They are mostly against AI in a nebulous sense, but even non-technical people use ChatGPT or equivalent. I think that the critical mass is already there and the tools are convenient enough that they couldn’t outlaw them without a massive uproar.
Detection tools don’t seem all that relevant to mainstream audiences’ use of LLMs. AI companies will sell this as a safeguard against misuse, and everyone will be happy about it. The politicians will say they accomplished something, the AI companies will slowly turn public opinion, and the public will have shiny toys.
> The number of legitimate use cases for costly frontier models drops significantly once eliminating professional jobs is off the table.
I don’t know. They can open possibilities that we don’t necessarily consider.
One example I have is a friend who is getting his house refurbished. He’s not an engineer or a material scientist. He does not have enough free time to read thoroughly on the many subjects involved. With a decent LLM, he could untangle the technical documents sent by the architect and the contractors to really understand what was going on and be involved, rather than passively follow the architect’s advice. For starters, the LLM was very useful in finding issues in the quotes he received when he was looking for an architect. Those were long, technical documents, with no really standardised structure and full of jargon. I don’t think that person is going to want to stop using LLMs now. Many people are having this sort of moments right now.
> In the near future the EU will likely come down with heavy intervention to prevent AI from impacting employment rates across Europe.
Maybe. But i don’t believe the EU is well equipped for that. Labour laws are largely local and different in each member state. The EU regulations are basically the common denominator, ore or less, and it is easy to see why: for regulations to get adopted, they need a strong enough majority in the Commission, in the Parliament, and in the Council. It is very difficult to get anything controversial that affect the sovereignty of member states passed.
The angle of the current AI regulations is that they set the rules for the single market, which is where the EU is the most legitimate. It is difficult to see market angle for the effect of AI in labour, and I think enough member states would be keen to kill the project.
Also, there are many influences at play, but the EU is fundamentally an economically liberal institution. It very rarely goes in the direction that reduces economic activity. Look at how clumsy it is at fighting against cheap Chinese imports. I don’t think the institutions themselves would really want to make AI illegal. Companies have too much to lose.
Great points all around, lots to think about. It's totally possible that I am way off in my assessment of the future.
I think your comment about torrenting touches on an interestingly relevant case study, specifically media piracy. The seed of that technology was sown when the internet was still just clusters of machines passing files around and then exploded with the PC, Napster, and TPB. Napster tried to be a legitimate commercial enterprise with a business model of undermining the ability of copyright holders to rent-seek on the consumption of the material they 'owned'. At the core it was a novel technology (P2P) that revealed an economic arrangement to be out of date (if a distributor no longer has to manufacture a copy of the media for each individual consumer, their business model boils down to rent-seeking). Western legal systems were quick to rule on the matter (in favor of copyright holders), and I have no doubt that they were looking ahead to a future wherein media creation was totally disincentivized by said novel technology.
Now a novel technology (the cloud inference-backed LLM) is challenging another economic arrangement. This time around, the arrangement being challenged is the higher education->professional job pipeline. All advertising and media messaging aside, it really does seem like the frontier labs are only economically viable if they get massive enterprise deals across a broad spectrum of industry. There is fundamentally one chunk of capital organizations are going to spend either supporting their talent pipeline or padding it (to put it gently) with enterprise LLM deals. If the latter path is taken too far, consumer spending plummets (due to lack of middle-class incomes), assets backed by consumer debt/spending fail longterm, and we will have to deal with a deluge of socio-political issues stemming from the absence of real social mobility (we are in the early stages of this now, incidentally).
All this to say, I think parallel situations from the past can guide our thinking re: AI regulation and the forces shaping it. Reasoning based on regular political/economic incentive structures (e.g. economic liberalism not wanting to reduce economic activity) will fly out the window at lightspeed once fear becomes a factor. I personally think its great that LLMs empower individuals such as your friend to increase the control they have over real issues in their life; that is what technology should be doing for us. Cloud inference-backed LLMs are doing the opposite: drastically reducing the power that the everyman has over his socio-economic future by throwing high-paying career paths for a whirl and incentivizing powerful organizations to destabilize the labor market. Only time will tell how this plays out.
What happens if we train models (GPT or human students) using the outputs of a model with text havingbthose watermarks? Is there something preventing the watermark from being learnable?
In addition to holding the key, wouldn't you additionally need to know exactly which model to check against? So for passive detection to happen, I think each company would need to check every message against every model version? Also would need to spend resources re-invoking the each model version against each message.
Probably they bias the RNG for selecting the next token. This can be done practically in a lot of ways, including during training.
I suspect the signal will be significantly under the noise floor, so it's not detectable if you don't know exactly what to look for, but certainly you can submit more information then the textual contents.
But if the user's prompt is in the context, you don't know exactly what the RNG chooses between. I don't know what trick they use to get past that, but it seems impossible to get by it in the general case (i.e. if the prompt can be anything) and you'll probably quickly compromise quality if you try.
A reasonable guess about the algorithm is 'A Watermark for Large Language Models' (https://arxiv.org/abs/2301.10226). The idea is that each generated token (or bigram) seeds a strong PRNG that splits the vocabulary into a 'green' and 'red' set. The sampler then tries to select a 'green' next-token for generation.
After-the-fact checking only needs the vocabulary splitter, which is independent of the LLM. Over a sufficiently large text non-watermarked text would expect to use green and red tokens with the baseline probability, and that difference can easily become statistically significant over sufficiently long texts.
The basic algorithm has obvious knobs to tune, among them the initial ratio of red to green tokens and how hard the sampler tries to pick a green token. These would balance fidelity to the original distribution against watermark detectability (minimum required content length for statistical power).
Anthropic actually tells you the approach they use, and it's not that. From their Claude Text Watermark page[0]: "Claude’s text watermark is a version of the SynthID-Text approach published by Google DeepMind in a Nature paper in 2024."
The Nature paper is "Scalable watermarking for identifying large language model outputs"[1]. This method does not separate out tokens into separate classes, but merely uses a seed for the PRNG that selects which among the most likely tokens generated by the LLM will actually be output. This has the advantage that there's no green and red token sets, so no token is systematically favored or disfavored. If a particular token is overwhelmingly predicted to be the most likely candidate, it will almost certainly be selected, so the watermark doesn't affect that. Even if there are several choices of output token at a point that have similar probability of selection, the watermark doesn't systematically bias in favor of one token or the other.
This is actually a quite elegant method of watermarking that, contrary to people's fears, won't adversely affect the model output. The main concern I have with it is that it appears that you can't actually test the watermark locally, without uploading it to Anthropic. I'm not sure why that's the case, since there's no particular reason the watermarking key has to be private, except if you want to prevent others from generating text with their own LLMs that is watermarked to look like it's generated by Anthropic - but everybody wants their text to not have the watermark.
Simple version: In instances wherein the otherwise statistically chosen next word is a "toss-up", watermarking removes the randomness by imposing specific choices, determined by a key. This then becomes a detectable pattern when scanned with the key (stastically—detection itself is probabilistic).
>use it to store arbitrary information
No additional data is embedded. The range of available data is constrained by the text being generated (i.e. the sets of "next words" per text).
From what I’ve read, they won’t be imposing specific choices, but using a different (biased) RNG for those “toss-up” choices. With enough sampling, you could detect if the RNG was biased or not.
This is what I meant by "imposing specific choices, determined by a key". Maybe "impose" or "specific" were too strong in my attempt to simplify?
I attempted to clarify that the impositions themselves are not deterministic, by indicating that the entire process is still probabilistic.
Maybe Anthropic's explanation is simple enough [0]:
>When watermarking is used, choices are still made at random, but the source of the randomness is different. Instead of using an arbitrary random number generator to pick the next word, watermarking uses the key and a few words that come before to settle what word the model should pick. That is, the words that Claude picks are still random, but now, one can check the sequence of words and see if it’s consistent with the choices Claude would make if it was using the key. If it is, one can assign a probability that the text was generated by Claude.
If it stores unique information, by definition it can store arbitrary information because it can point to arbitrary information. So they can have it relate to anything they want. Even a full breakdown of the original text if they choose.
Removing the watermark usefulness depends on your use case.
If you care to avoid detection, yes, it is useful. If you care about the best possible sequence of words, then the damage is already done once watermarked.
Is watermarking really watermarking if it can’t be independently verified?
I mean, all of these text content watermarking schemes require the company to assess if the text was AI generated or not. They aren’t going to tell us where the toss-up tokens are or what is in the red vs green pools of words.
Evidence is a massive problem here. As well as the extremely high threshold for US defamation; political candidates routinely tell the most absurd lies about each other.
The high threshold is only for public figures, which must show actual malice. For everyone else (e.g. your typical student falsely accused of plagiarism), they only need to show the statement was false, published, targeted the plaintiff, and caused damage - it's still defamation even if they were simply negligent in checking the facts. The company might be able to cover themselves if they are very clear they are just giving a statistical probability and not saying their model was definitely used, but if they imply they have hidden information that points to their conclusion (which is kind of the whole point of their algorithm) then that's not a valid defense. I can't say they would lose in practice, but in theory they ought to lose.
But not insult, and even for defamation, you will be prosecuted and have to prove the statement is true, even if the politician and everyone else knows it's true.
I think this is a very real concern. But I’m not sure of any way around it. Any stenographic system that you have the code for can be trivially defeated.
I wonder if this would be a good use for homeomorphic encryption. There might
be a way to let anthropic check some text without actually giving them access to the source text. Any experts around? We could use your skills!
> Any stenographic system that you have the code for can be trivially defeated.
They're giving you an oracle regardless, which is almost as good. Take LLM output, make some modification, ask the detector if it's LLM output, repeat until you learn what kind of changes you have to make to defeat it.
Or don't even bother learning what to do, just make arbitrary changes until it says it's not, so when the person they're submitting to does the same check it says the same thing.
I assume this oracle will be behind 20 layers of anti-bot protection, CAPTCHAs and hardware attestation challenged. It will be incredibly painful to use. It won't stop the motivated attackers, but will make it too annoying for the average person.
They could (..and probably will..) store that version and then refuse the check if this attack is detected, i.e. the version is too close to a known LLM output.
Alternatively they could also just keep saying "yes" if it's close enough to a version that was close enough.. Although that would enable the attack to allow arbitrary text to be "proven" AI, by slowly morphing close-enough generated material to the desired text. But perhaps this is not a problem they are not concerned with.
To satisfy the letter of the law I expect it's enough to just provide the oracle, without any mitigations.
Won't we just be able to fine tune OSS models to detect these patterns across providers? It will be cat-and-mouse but my bet is it converges to a central detector that isn't affiliated with any model provider.
> Won't we just be able to fine tune OSS models to detect these patterns across providers?
A good fingerprint should make use of cryptographic signatures. Without knowing the keys, the fingerprint should be indistinguishable from noise (or just random token selection)
>Any university using AI detection in their submission pipeline, or lawyers, editorialists, proofreaders that check for AI marks will be sending significant amounts of text like unpublished research, books, potentially internal documents and more, most of which is high quality human written, to dozens of AI companies, blindly trusting they won't train on any of that.
Uh, yeah, why do you think it's setup this way? The frontier companies desperately desire more high quality human text and this is how they are planning to get it for free.
And I'm sure students will use tools to have every other paragraph written in the style of a different AI, in an attempt to defeat this fingerprinting.
I saw that someone already created a Github repo for a Python script that strips the watermark out of Claude generated text. It was released, I think, within 24 hours of the announcement. I cannot attest to how well it works, but I found it humorous nevertheless.
How would that change anything? The proposed watermark is applied while the output tokens are being chosen, taking that text and running it through an LLM again would just repeat the process.
You take the output and run it through another LLM with "please re-write this in xxx style". Then you repeat that a few times on different part of text and glue it all together at the end.
Exactly, so even if you are avoiding AI, any interaction with society is now being structured so you have to submit to the digital surveillance equivalent of a cavity-search machine. What a dystopia awaits the budding generations.
Anthropic had absolutely nothing to do with this. The Chinese models will soon be adopting such devices as well. It is an overwhelming force coming inter alia from educators worldwide.
I can't speak for Anthropic, but with Google's SynthID, the algorithm is actually public. However, checking (or creating) the watermark requires a symmetric key, and the providers likely wouldn't share that key.
> And even that is not sufficient, as the text might have been generated with ChatGPT, Gemini, Grok, MistraL
I would think that these things would eventually converge and we’d get one watermarking algorithm as an industry standard. That way, all major provider would follow it and we’d get independent software for checking. This would partly limit the efficacy of the watermarks, but on the other hand if it’s done correctly, removing the mark could still be enough of a pain that casual users would not bother. That would obviously depend on a lot of factors. It would at least add significant friction in the production of daily slop.
Of course it wouldn’t do much for thing like foreign propaganda but that’s a whole other discussion we need to be having.
> Any university using AI detection in their submission pipeline, or lawyers, editorialists, proofreaders that check for AI marks will be sending significant amounts of text like unpublished research, books, potentially internal documents and more, most of which is high quality human written, to dozens of AI companies, blindly trusting they won't train on any of that.
Isn’t it already what they are doing right now with some of the plagiarism detection tools? Not every university is going to have a representative corpus, and yet they are all using the software. So I guess the provider is doing the work of feeding all that data to their algorithm.
You can’t check it with only the algorithm, you need the secret seed key. Which will be different for each provider (and they’ll probably have and use multiple). And you need the llm itself, to generate the potential tokens at each step.
I’m worried that until such tech is perfected, pervasive and uniform across all models, education will be dead, as it certainly is at the moment. Flat out dead.
I read stacks of term papers all year and it is a reality that, apart from such schemes, we are in an extinction event for civilization.
Education is in the best place ever for people who actually want to learn, they can have 24/7 access to a tutor with a wide breadth of knowledge and infinite patience for stupid questions. Education is in a terrible place for people who just want to get a degree and don't care about actual learning, but such people generally don't contribute much anyway, and are the easiest to replace with AI, so no big loss.
its lovely training data. no detection? add to training set -_-.
its also kind of laughable that somehow people are trying to prevent the outputs not to be altered. Asif you cannot manually paraphrase anything you can read. So the only solution would be, to make it utterly unreadable (which is not possible, it obviously defeats the purpose of the thing).
Why not switch it around? Solves problems of privacy (no data upload), is far more doable and it's far more important to be able to verify content has not been tampered with after originating from a human or capture device (like a camera). We also have decades of cryptographic experience in reliably signing text, images, video, etc. and it doesn't break apart because a text is too short.
Having proof that content (especially images and video evidence) is unmodified (whether via Photoshop, Paint or a model) is far more valuable then having evidence that an image was manipulated or generated fully by a model (which still leaves other forms of manipulation), I feel the same goes for human authored vs generated text. Free to admit that using models to generate any kind of media whole-cloth is still unappealing to me and I still pay for commissioned artwork or make it with my limited abilities for what that's worth. Do like to (poorly) write my musings too and see UX as something were thoughtful contributors (like the opinionated, sometimes controversial, but certainly talented GNOME Gitlab contributors) can make a major impact.
Code can be beautiful, interesting and serve purpose beyond execution, of course, but for most people, in most cases, it does not in the same way as audiovisual content (not limited to art). Having code just to execute and resolve a problem can have value all in itself, the code being a means to an end whose quality, let us be honest, was barely a concern in most corporations long before LLMs.
Also have rarely (honestly never) before LLMs fully owned all parts of any code base, always relied in part on someone's prior effort in (Flutter/Dart mostly) packages, whereas when writing, drawing, etc. I have far more situations where I make something from scratch and everything there is only there because of my conscious decision. Even simple marketing mockups that, quality wise, any modern model would beat feel different when I was fully in control, where to place what, etc. Objectively worse (at my skill level), probably, but still never the same.
Knowing something was made from scratch by a human has value to me, beyond misinformation prevention. Knowing for a fact that LLMs were used instead of importing a library, using a template, or something similar that leads to expending similar amounts of effort, I don't see that being nearly as valuable. Heck, with all the importing and my experience back then vs now, I am spending more effort actually fully reading any LLM output in my code then I spent back then auditing Flutter/Dart packages. Then again, LLM output fails far more unpredictable then those messy packages that simply got Gradle to take down my system...
Happy to admit, I have been skeptical of watermarking LLM output being feasible for quite some time and having looked into SynthID Text and proposals being researched, I am convinced that it is challenging to impossible beyond the lowest common denominator and less important then proofing human authorship.
It will catch people just copying LLM output into their replies without thought, which is not a negative in my book, especially if it is not discernibly affecting output quality in regular use cases. Anyone who wouldn't copy Wikipedia into their dissertation will, in my opinion, be able to bypass text watermarking as proposed however, I feel we need to be honest there.
Thing is, if that's the case and text watermarking will only ever catch LLM created slop, is that a bigger problem then the misinformation, harm to creators due to authorship questions and accusations, making it harder to use evidence in proceedings, teachers not trusting students even when they did the work themselves, etc.? Signatures for all such cases will be difficult to implement, yes, but I feel are going to be of greater value in the not to distant future and I equally feel are not impossible, not least because idiots will always want to hide their LLM usage, whereas human authorship is something they take pride in and want to proof.
Fair point, for text it is far harder to prevent signatures being applied to generated text vs images at the moment of capture and most approaches I can come up with to remedy this can either be bypassed (edit histories can be output by models similar to humans) or will be controversial. Taking a page out of the anti-cheat textbook, mainly written for gaming, there are methods which might hold in the medium term. Less a fan of kernel level myself, though it might be worth exploring as there has been massive investment by the games industry into making it somewhat robust, but the approach Valve has taken with VACnet could be an inspiration worth exploring that is less invasive into peoples systems. Keystroke analysis, etc. could be relied upon as a basis for signatures, harder to spoof for current day LLMs over generating edit histories.
I will fully admit that at a point in the future, maybe not too soon, models may be trained to bypass that too, at which point we are back where we started. As a skeptic of the extend that capabilities are emergent in LLMs vs specific to training data, I am somewhat hopeful that unless models are specifically trained for evading such human detection solutions, they'd struggle to do so, but it could still end up as a byproduct of improved, lower latency computer use focused training. Not emergent as the term is used in regard to models because that is still output performance improvements clearly traceable to very specific training data, but incidental as the goal of said training data was not to bypass.
For what it's worth, I find human authorship being verifiable to simply be the more crucial problem over watermarking model output, so if research is to focus on one, I'd rather it the former. Maybe both signing human authored content and watermarking LLM output are both only possible in the near term, I hope not but fear it that might be the case. If so, we as a society will have some major challenges ahead (beyond all the ones we'd have anyways).
Alternatively, we could also just start scanning everyones eyeballs...
I've been thinking for a while that all of this is just trying to grasp tighter the last bits of sand escaping between our fingers. The end game, perhaps, is trust. Do you trust or know the source? If you don't, assume it was AI generated. If you do, accept it as authentic based on whatever they disclose, but know that it's possible they aren't being totally honest or were themselves fooled in some way, depending on the context.
Then build our assumptions and how we operate around those trust levels in the digital realm.
Well, the end game is that AI is better at thinking us, and having a human brain involved is a net negative. The goal is to let the AI do the toil if thinking for us, and we can get the rewards.
Thank you for stating clearly situation. I fully agree with your assessment. For almost 10 years now I have been saying that we need to virtually watermark reality. By "virtual" I mean store the metadata about the digital capture on a public blockchain. Then my devices could have a built-in "fake vs real" detector.
Artists, photographers, journalist, etc. are going to want and need this.
Some sort of signatures to verify legitimate unmodified photographs (taken with a relatively secure device, e.g an iPhone) seems like a reasonable idea
But even then, people will be able to point that camera at a manipulated/generated image (either printed or on a screen). Maybe that one could be solved if the photo included some depth information?
a lot of cameras do have an in camera hardware based cryptographic signing (i think it runs off its own chip on them?) but any modifications to the image immediately break that. so for stuff like journalism etc where youd hopefully have a lot of settings dialed in on camera for quick turn around would keep that fingerprint intact.
> blindly trusting they won't train on any of that
being allowed to train on any data that you can legally obtain ought to be a right for anyone.
After all, i am allowed to learn off anything i can legally read (and perhaps even illegally read). The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.
It's a bit different when "training on any data" means basically storing a lossily-compressed copy of that data, that could be spit out years later if the model decides to do so.
It's exactly the same problem as with humans, though.
It's part of why we sign NDAs, and why their duration is measured in years (and that's not even targeting the human retention - just duration after which information ages enough that its disclosure is not likely to negatively impact anyone who cares).
It's not exactly the same problem, in that you can parallelize usage of an LLM and copy it over to another computer, but cannot do the same things with a brain. Put it another way, humans do not have the processing power needed to answer hundreds of millions of queries per day, while LLMs do.
> being allowed to train on any data that you can legally obtain ought to be a right for anyone.
I have the opposit viewpoint to the extreme. They shouldn't be allowed to even read that data until they are very clear about what they will or not do with it.
Can they publish it? Can they store it? Can they use the information in it on prediction markets? Etc.
Humans reading texts historically come with little negative consequences, but machines reading and processing texts en masse is more dangerous and should be regulated.
> Humans reading texts historically come with little negative consequences
IDK, we do have laws against opening other people's mail. Those have been on the books for hundreds of years. Seems like someone figured out a while ago that certain unauthorized humans reading certain restricted text wouldn't be good.
> Humans reading texts historically come with little negative consequences, but machines reading and processing texts en masse is more dangerous and should be regulated.
> After all, i am allowed to learn off anything i can legally read (and perhaps even illegally read).
Are you a tool?
Because humans gets rights, tools don't.
Arguing that untrained or partially trained models should have have rights is a different argument to arguing that a trained model should get the same rights as a human.
At the risk of stating the obvious, there are a lot of legal rights that are human-specific (voting, holding office, filling lawsuits, etc.). It's not at all obvious why you think that you as a human being legally allowed to learn from something implies that it should be legal to train an LLM on.
> The only thing not allowed (rightly so) is to produce a copy with enough similarities that it can be replacing the original.
But LLMs are replacing the original, just in different words.
And what does 'legally obtain' mean in this context? Copyrighted content is usually licensed for specific purposes. So if a license is given from training your LLM, then by all means do! But what if the license is 'for personal use'... ?
Oh so if I use mickey mouse in a completely original production that doesn't replace the existing work by Walt Disney, you reckon they'll be fine with that?
> You are one person. The corporation is not. Scale matters
Correct, if you violate it too often to count, you have to pay around less than ~2.5ct per violation.
So the lesson here is: Create a company to do torrenting professionally, and resell its values for higher prices. Then get sued and pay a dime on the dollar you made.
Copyright (or any other such restriction on free use of information) creates power for owners by the simple fact that it turns information into something that can be owned.
What shines through here is that the AI writing / reasoning this is about as lazy as always, declaring a hard limitation where some engineering might solve or work around the limitations. In modern JavaScript a surprising amount of tasks can be handled, e.g. with Workers and in a streaming way.
Beer CSS is great. I've used it for multiple simple projects and it provides a great DX with the clean html code and the many snippets on the official website. The only downside is that LLMs are quite bad at working with it from my experience, maybe it's just too simple for them..
Yeah, spot on. I had an agent delete some files it shouldn't have as well, similarly to me making the same mistake. I think system prompts should default to using `trash` over `rm`.
For now that's just in my AGENTS.md, and gets honored most of the time.
You can always use something like this [1], which will make sure any file removed on the command line via rm (or other utilities, like git rm) ends up in the trash instead
Amazing observation, and I'm certainly guilty of it too, but it is just way too convenient not to sandbox it, and some tasks right away depend on not being sandboxed.
For anything other than writing code directly in a fully contained git project, where sandboxing might work well, it requires access to system wide tools, user configuration and more.
Occasionally I tell the agent to do everything inside of docker, which works too and it leaves the system alone then mostly, but adds significant overhead and slightly degraded perceived quality / effectiveness.
I think the most important takeaways are to have reliable backup strategies, access control and security mechanisms, which is a win regardless.
Whether by the agent or the human, mistakes happen (like a rm -rf * ran in the wrong directory), and where they would be devastating, there should be other protections than just "hope it won't happen" or "rely on a sandbox to prevent agent error".
> made with <3 by Ayden
That's the cherry on the cake - indicating genuine effort and attention to detail put into something, yet not even checking the credit at the bottom.
reply