Building a website in 2005: "here are a list of tips to help crawlers index your website"
2025: 'here are a list of tips to help avoid crawlers indexing your website"
Building a website in 2005: "here are a list of tips to help crawlers index your website"
2025: 'here are a list of tips to help avoid crawlers indexing your website"
this toot brought to you by the ever increasing list of user agents my sites serve HTTP 402 Payment Required
If you have a fediverse account, you can quote this note from your own instance. Search https://hacky.town/users/rtyler/statuses/115681346312072733 on your instance and quote it. (Note that quoting is not supported in Mastodon.)