AI Crawler Guide — manage GPTBot, ClaudeBot and others

Complete overview of AI crawlers (GPTBot, Perplexity, ClaudeBot, Google-Extended, ...): what they do with your site and how to configure robots.txt.