Docs menu

Bot Access Checks

Most WordPress sites quietly block AI bots in robots.txt or via security plugins — sometimes by default. We test the live response for the bots agents actually use and report exactly which ones are allowed.

B1

Explicit AI bot rules

weight 8

We probe these user agents and grade based on which are allowed:

User-agent	Operator	Used by
`GPTBot`	OpenAI	ChatGPT training & browsing
`OAI-SearchBot`	OpenAI	ChatGPT Search results
`ClaudeBot`	Anthropic	Claude training
`Claude-Web`	Anthropic	Claude live browsing
`PerplexityBot`	Perplexity	Answer engine
`Google-Extended`	Google	Gemini & AI Overviews

How to fix

robots.txt
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

B2

/llms.txt roadmap

weight 5

An optional /llms.txt at the site root gives AI agents a curated map of your most important pages — far more useful than a 10,000-URL sitemap when an agent only has 8K tokens of context.

How to fix

markdown
# Your Site Name

> One-line description of what the site is about.

## Pages
- [About](https://yoursite.com/about): What the company does.
- [Pricing](https://yoursite.com/pricing): Plans and costs.
- [Docs](https://yoursite.com/docs): Product documentation.

B3

Content Signals (Cloudflare)

weight 3

If your site sits behind Cloudflare, you can declare AI usage policy with the new Content Signals headers (ai-train, ai-input, search).

Optional check

This check is info-only. It can earn bonus points but can never drag your score down.