> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience.
Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a series of programs actually but let’s not get into the details)
One possible way to do this: rather than simply burning hashes for nothing, Anubis can burn hashes to mine crypto. This way, the bots pay for the infrastructure they consume, at least in part.
Combine with heuristics like in the article, so you can crank up the difficulty only for likely bot traffic, and you've got a bot supported site.
CPU crypto mining stopped being viable over a decade ago. If you want a client to pay for the infra cost of their request, they're probably going to need to work for hours to do so.
> Anubis wants "slow for bots but fast for humans"
That's not true. The goal is to raise the cost for bulk scrapers to the extent possible while minimizing any inconvenience to regular users. It's not about being slow for bots but rather reducing the asymmetry of the problem (from the perspective of the server) by imposing a (minor) cost on anyone requesting a page.
That's a better way to put the goal. But does it actually raise the cost or reduce asymmetry in a meaningful way? If it makes crawling 0-5% slower then it's really not worth the annoyance.
I know having the Anubis javascript at all gets in the way of some bots. But is the calculation helping enough?
> does it actually raise the cost or reduce asymmetry in a meaningful way?
It's complicated. I think anubis currently only collects low hanging fruit so to speak but in practice that's the bulk of what was causing problems so it's good enough.
PoW in general absolutely yes - I think monero clearly demonstrates this. Last I checked it remained viable to mine monero on at least somewhat low end and at least somewhat outdated consumer CPUs. It seems to me that fact implies that a well constructed PoW scheme can in practice squash the impact of any performance differences to within at most a couple orders of magnitude.
Sure. Even if PoW in general is sound, when applied to scraping it's always going to be a matter of what the client is willing to "pay". It's entirely within the realm of possibility that a scraper could be willing to "outbid" the user of a 5 year old tablet.
But it's also true that charging anything is going to be expected to reduce server load, that the cheaper something is the more carelessly people will tend to make use of it, and that at least in general bot operators are expected to be more cost sensitive than regular visitors on account of scale.
To that last point consider that a regular visitor will generally have an idle CPU and multiple gigabytes of free RAM. Meanwhile a competent scraper will generally be optimizing request parallelism per cheap VPS while an incompetent one will just point and shoot (and proceed to blindly visit every cgit url). The latter is by far the most troublesome and conveniently also the most likely to be impacted by even a poorly implemented PoW scheme.
I keep seeing comments along the lines of anubis only working because it bounces bots running lightweight setups without js but stop and ask yourself why are they running such lightweight setups to begin with?
> why are they running such lightweight setups to begin with?
It's easier and still gives access to almost everything. I don't think CPU limitations are the main issue there. As far as computer resources, I'd bet RAM is the bottleneck to running the script, not cycles. That suggests a RAM-heavy hash might do better than SHA256, but still agrees with the idea that the PoW in Anubis is not the important part.
> To that last point consider that a regular visitor will generally have an idle CPU and multiple gigabytes of free RAM. Meanwhile a competent scraper will generally be optimizing request parallelism per cheap VPS while an incompetent one will just point and shoot (and proceed to blindly visit every cgit url). The latter is by far the most troublesome and conveniently also the most likely to be impacted by even a poorly implemented PoW scheme.
I considered that. But while a visitor is idle they need the result quickly and they face the full brunt of the challenge all at once. A scraper can be working on a thousand sites at once so the per-request impact of the challenges is a lot lower, and patiently working through challenges in the background usually reduces the real cost, potentially to almost zero.
And you'd better not need multiple gigabytes of RAM to visit your site, that'll freeze my computer.
Odin solves this by letting you declare all arrays as either normal (AoS) or SoA, which helps with this use case a lot. Otherwise very similar ideas, except all the memory goodies. Underrated language!
Well, the point of me not revealing a title is so that if you run across this book later, you won't think, "OH! This is the one where Pavel said X happens!"
A spoiler by itself, without a link to the book, isn't a spoiler at all, which is why I didn't include the title.
If you're curious, I can drop five recommendations for good books, one of which is the work I'm referring to :)
I suspect the technical interview devolves into an act of shunting the blame of potential bad hires away at some places. Elite credentials and leetcode shibboleths serve the same purpose: this person has checked the boxes we've agreed upon as important in the hiring process, regardless of the job duties. If they turn out to be a bad hire, who could've known, esp. with that Ivy League bachelor's degree?
It's less the tree operations themselves and more that this candidate just didn't know the shibboleth, so they shouldn't be allowed in. Cue the vague feeling of disgust on the interviewer's part that the interviewee didn't know something "so simple" in a tense environment with lopsided power dynamics.
>shibboleth
Fantastic word right there, had to look it up.
That's exactly how I'd describe the hiring process these days. Relevant expertise seems like its only seen as a bonus. That or I'm interviewing at some terrible companies.
The role of such standardized coding interviews is to ensure consistency when hiring at very large scale. Some big tech companies hire thousands of people a year, and interview ten times that.
Candidates have a very diverse set of backgrounds, age, experience, culture, education, interests, experience. The coding interview is a common denominator: undergrad algorithmic knowledge, being able to solve problems which are application of classic algorithms. This is something that a decent coder should be able to prepare for within a few months of regular practice. In my case, I can tell that these months of practice were the best investment in my life. And honestly, I feel I'm a better programmer after this preparation.
> Cue the vague feeling of disgust on the interviewer's part that the interviewee didn't know something "so simple" in a tense environment with lopsided power dynamics.
This is your interpretation of the situation. As an interviewer, I want the interviewee to succeed, I understand it's stressful for them as I've been in their shoes before.
Ultimately, there are more candidates than positions, so whatever the interview is, people will complain about it. I think leetcode is a decent interview technique. I wish we would give more time to candidates though.
You could also solve this with competition. If there were 10 ISPs it would be disadvantageous to give your customers reasons to leave you. Why aren’t there more ISPs? Maybe too many regulations. It is trivial to lay cable, except of course all the permits.
Or maybe it's an oligopoly where the incumbents have carved up the market and stopped competing, milking their customers instead.
Broadband is then extra special if you let the ISP also own the infrastructure as everyone has to reconnect their service to every house instead of one company (or forbid, the govt) owning the pipes and several companies competing for providing services over those shared pipes.
Imo the competition model doesn't necessarily (always) work that well for infra.
Because it's illegal to dig up the road without a permit and they won't give a permit to install new fiber when the road is already full of perfectly good unused fiber. They only grant one of those the first time.
Also old people. They need younger people to support them, like grow their food, fix their houses, etc. But given that I plan on growing old, as I hope you do too, we should make sure the generations after us are capable of taking care of us.
The current best solution is written by Fabrice Bellard, probably more famous for his other works like FFMPEG, QEMU and TCC - https://bellard.org/. Check him, pretty much everything coming from him is pure gold.
When people say there are no 10x programmer, I name Fabrice Bellard. Him being a 100x programmer invalidates the logic of those trying to explain there aren't 10x programmers.
Interestingly, this is the first time I'm seeing apple recommend using neovim+terminal for development work. They even showed that LSP integration is supported.
Haven't they been announcing Pros and Max's around December? I don't remember. If they're debuting them at WWDC I'll definitely upgrade my M1. I don't even feel the need to, but it's been 2.5 years.
Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a series of programs actually but let’s not get into the details)
reply