Speaking from the scraper’s perspective, I like proof of work; a ten year old 96-core server will cost a couple of quid to run for a few hours and will grab an absurd number of pages thanks to the access granted by repeatedly solving proofs of work. Small slick codebases too!
There's also the Anubis idea where your PoW is persistent until your IP address or session cookie changes, so you get to skip PoW in exchange for making yourself identifiable, which means the PoW can then be ramped up to take a couple of minutes.
I don't use Anubis though. I just make my site not take five seconds to render a page so bots can overload it easily? It's not actually that hard?
I think we're talking about 2 different things. PoW is annoying for basic scrapers but it really doesn't affect enterprise grade bot operations with access to unlimited residential proxies.
look, if you have the kind of bank to pay $1/GB of web pages to get to whatever we're hosting, maybe just send an email to sales so we can give you discount on that.
Depends on what type of scraping you're trying to stop. For the dumb scrapers that would try to scrape every page on a git forge (for which there are a bazillion pages for a modest project, because of how the site works), yeah it might deter them enough to stop. For anything high value (eg. reddit comments or retail prices), 10s of cpu time isn't going to stop them.
It will not scare away bots but 10 seconds of wait (CPU or only a sleep) will turn away many real users. "This site is so slow, I'll use something else." A kind of reverse captcha.
Sure, the whole premise is exactly that proof of work reduces the value of scraping, while having negligible impact on users. If the data is so valuable that bot operators are willing to pay 10s of cpu, then other measures are necessary.
Nevertheless even for these high value cases, you can still argue that it disincentivizes the business model, it becomes less efficient.
If it's high value, there isn't really much you can do that will be completely effective. Traditional captchas can often be beaten by AI, or by "captcha farms" where impoverished people are paid pennies to complete captchas. Fingerprinting can be beaten by using a full browser to make the requests. Basically anything you do is just a matter of making it more expensive for bots to access it.
Beating fingerprinting and beating traditional captcha is far more expensive than solving pow. Pow doesn't stop anyone, not even the most novice bot operators
There are other captcha alternatives like Turnstile, for example Private Captcha, Altcha etc. - they are owned by mostly “small” independent companies, they are not visual captchas (proof-of-work based) and very accesssible.
I remembered $30 from some comment I read, but didn't look for it later. If it was yours, thank you! (def. thank you for the Wallmart link! - would you like a credit in the blogpost like a quote?
>Excuse me, what do you mean there? The author happens to read HN too.
Read the rest of the comment. It's not suspect because it's referencing HN, it's suspect because of the way it's referencing HN. Specifically, its use of the phrase "the HN thread", even though it wasn't mentioned before. Maybe it's a editing gaff, but it's also consistent with how an LLM would write if presented with a list of sources.
Yep, this feels like a smoking gun. The others are circumstantial, maybe indicative, maybe not. While there’s a chance this is an editing gaff, its overwhelmingly likely to be LLM, ahem, “cruft”.
Have you seen Private Captcha? It's main point is privacy and GDPR compliance (while being self-hosted). It's a more modern PoW captcha without the limits of Altcha (mCaptcha is mostly out of equation at this point).
1. At it's core, it's a Proof-of-Work captcha, so compute load depends on the usage pattern. AI (LLMs) does get better at image/audio recognition, but compute task is the same anywhere, does not matter who "clicks" the checkbox.
2. Bots usually have such access patterns that cause compute task difficulty to become prohibitively high, requiring exponentially more CPU resources. This renders bots economically nonviable for submitting forms or scraping website.
