Can the AI tools that search the web read your site?
Why it matters, sources and review
Your robots.txt tells crawlers what they may read. If it shuts out the search crawler behind an assistant, that assistant cannot cite you.
OpenAI, crawler overview: OpenAI names its own crawlers there, so it is the place to confirm which names robots.txt has to address.
- Evidence
- AI product
- Sources
- OpenAI, crawler overview, undatedA
- Anthropic Help Center, crawler controls, 2026-04-07A
- Perplexity, crawlers, undatedA
- Meta, web crawlers, undatedA
- Apple, About Applebot, 2026-09-04A
- Review by
- 2027-04-18
- Was
- AIR-1.1 in Index 11
- Full entry
- Detection, bands and fix, in the standard