IETF just setup a new mailing list "Web-bot-auth": https://mailman3.ietf.org/mailman3/lists/web-bot-auth.ietf.org/
"Currently, wide practice is for Web sites to identify non-browser clients using IP addresses, the User-Agent header field, and/or reverse DNS. All of these techniques have limitations and deficiencies, and at the same time the need for stronger identity for bots is becoming stronger, as non-browser traffic on the Web grows in volume and importance."