Often you can't pick apart the good traffic from the bad.
Fortunately bots are usually pretty stupid. If you can outrun them on bandwidth, then change /victimpage.html to 302 to /victimpage-new.html. The web server or load balancer can send those redirects really fast and it doesn't take much bandwidth either. I have never seen a bot chase that redirect.
After a particularly nasty DDOS attack (where our upstream provider just shrugged their shoulders) I wrote an F5 iRule:
1. Check for the IAMNOTABOT cookie
2. If not there, redirect to /cookie-me?oldpage=the_page_you_were_trying_to_access
3. Set IAMNOTABOT=true cookie
4. Redirect to the old page
Another, similar, technique is checking for a NOBOT cookie and if it doesn't exist, serve up a page that uses javascript to set the cookie, then reloads.
This requires the bot to interpret javascript, and it can easily be configured in nginx and other load balancers.
Fortunately bots are usually pretty stupid. If you can outrun them on bandwidth, then change /victimpage.html to 302 to /victimpage-new.html. The web server or load balancer can send those redirects really fast and it doesn't take much bandwidth either. I have never seen a bot chase that redirect.
After a particularly nasty DDOS attack (where our upstream provider just shrugged their shoulders) I wrote an F5 iRule:
1. Check for the IAMNOTABOT cookie 2. If not there, redirect to /cookie-me?oldpage=the_page_you_were_trying_to_access 3. Set IAMNOTABOT=true cookie 4. Redirect to the old page