Follow

Does ahrefsbot just straight up ignore your robots.txt

@sir those types of crawlers usually have an option to ignore robots.txt when being set up.

@sir Maybe the server should just straight up ignore ahrefsbot. :)

Sign in to participate in the conversation
Mastodon

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!