https://sr.ht/robots.txt is a robots.txt | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

slipknotfan 3 months ago | parent | context | favorite | on: Ask HN: Releasing code under AGPLv3, but want to b...

https://sr.ht/robots.txt is a robots.txt

zionsati 3 months ago | [–]

Do LLM crawlers adide by this?

gus_massa 3 months ago | [–]

Ignoring that AI bot will ignore it, ...

Doesn't "User-agent: AwarioRssBot" needs it's own "Disallow: /" line?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact