WordPress guide
robots.txt in WordPress
Three reliable ways to ship the right rules — without breaking Google.
| Method | Where | Best for |
|---|---|---|
| Yoast SEO | Tools → File editor | Sites already using Yoast |
| Rank Math | General Settings → Edit robots.txt | Sites already using Rank Math |
| Static file | wp-root/robots.txt | Override any plugin output |
Copy-paste template
robots.txtUser-agent: * Allow: / Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / Sitemap: https://yoursite.com/sitemap.xml
Heads up: security plugins
Wordfence, iThemes Security, and similar plugins sometimes inject extra
Disallow lines or block AI user-agents at the firewall level. If a AI Grader by Magnetic WP scan shows bots are still blocked after you edit robots.txt, check your security plugin's bot-blocking page next.