Check whether AI search crawlers can actually reach your website
If the crawlers behind AI search cannot reach or read your pages, your own website cannot be cited. Crawlability checks access for AI search and search engine crawlers across your whole site, and gives your developer specific fixes.
brightlinesolar.example, entire website
46 pages checkedCrawler access
- OAI-SearchBot ChatGPT searchAllowed
- Claude-SearchBot Claude searchAllowed
- PerplexityBot PerplexityBlocked on 3 pages
- Googlebot Google Search and Google AIAllowed
- Bingbot Bing and CopilotAllowed
Disallow: /quote/ from the PerplexityBot group in robots.txt, if blocking those pages was not intended.What crawlability does
What it checks
For your whole website, or a single page, Crawlability reports:
- Published robots.txt rules for AI search crawlers, Googlebot and Bingbot
- HTTP status codes and redirects
- Canonical URLs and indexing directives
- Whether the page is in your sitemap
- Whether the main content is present before JavaScript runs
- Access challenges, such as bot protection, encountered during the check
Fixes your developer can apply
Findings come with the specific HTML or rule involved and a suggested change, to be applied only where it matches what the site owner intends. Run it again on an unchanged site and you’ll get the same findings. Each run is kept so you can compare.
Search crawlers and training crawlers are different
OpenAI documents OAI-SearchBot as the crawler that matters for appearing in ChatGPT search, separately from GPTBot, which is used for training. Blocking a training crawler is a legitimate choice and does not remove you from AI search, so Ansoar never presents it as a visibility problem. Our guide to which AI crawlers matter explains the difference.
A clear result means crawlers can reach your pages. It does not guarantee that an AI system will cite or recommend them.
Sources: OpenAI, Overview of OpenAI crawlers; Google Search Central, AI features and your website; Ahrefs, llms.txt study.
Common questions
Does blocking GPTBot stop my business appearing in ChatGPT?
No. OpenAI documents OAI-SearchBot as the crawler used to surface sites in ChatGPT search. GPTBot is used for training. They are separate decisions.
Do I need an llms.txt file?
There is no published evidence that an llms.txt file improves AI visibility, and Google says Google Search ignores it. Ansoar does not tell you that you need one.
Can it check sites built with JavaScript frameworks?
Yes. Crawlability checks whether meaningful content is present in the initial HTML, before JavaScript runs, and flags pages that depend on JavaScript to show their content.
How often should I run it?
Whenever your site changes, such as a redesign, a new section or a change to robots.txt. Each run is kept, so you can compare it with the last one.
Does a clear result mean AI will cite my pages?
No. A clear result means crawlers can reach and read your pages. Whether AI cites or recommends them depends on much more than access.
Related features
Try it on the jobs you want more of
Start a 7-day free trial, no credit card needed. Within about 10 minutes you’ll see how often AI recommends you and who it picks instead.
Rather ask something first?
Email ben@ansoar.io. It comes straight to Ben, the founder.