Today's learning experience: it looks like a HTTP Accept: of */* (or a missing one) from something claiming to be a browser is a good sign of a bad crawler, apart from bingbot (thanks Microsoft, or maybe bingbot is bad too). I learned this from https://kitsunemimi.pw/notes/posts/how-twitter-is-probably-crawling-the-internet-for-ai.html