AnsoarBot is the crawler used by Ansoar, a service that measures how AI assistants answer questions about a business. It fetches pages so we can tell our customers what their own site says and whether anything is stopping AI systems from reading it.
How to identify it
Every request sends this User-Agent header:
AnsoarBot/1.0 (+https://ansoar.io/bot)In robots.txt, which matches on the product token alone, the token is:
AnsoarBotIP addresses
We do not publish a fixed IP range, and you should not allow or block us by IP. AnsoarBot runs on cloud infrastructure with addresses that change without notice, so any list you built would break quietly. Match on the User-Agent header instead.
If your security policy requires an IP allowlist and a user-agent rule is not acceptable, contact us at hello@ansoar.io and we will work out what is possible for your setup.
Allowing AnsoarBot
Allowing it is only ever necessary if something on your side is refusing it. If our checks are working, there is nothing to do.
In robots.txt
If a rule in your robots.txt disallows us, either directly or through a wildcard, add:
User-agent: AnsoarBot
Allow: /In a CDN, WAF or bot-protection product
These usually match on the whole User-Agent header rather than the token, so add the full string:
AnsoarBot/1.0 (+https://ansoar.io/bot)AnsoarBot is not on any vendor's verified-bot list, so a rule that allows only verified bots will not cover it. It needs an explicit rule.
Blocking AnsoarBot
You are free to block it, and we will not work around a block. To stop it fetching your site, add:
User-agent: AnsoarBot
Disallow: /We obey robots.txt for our own fetches. If you block us, the customer whose workspace is checking your site will be told their page could not be read, and why.
What it does, and its limits
These are the limits the crawler is configured with, not aspirations:
- obeys
robots.txtfor its own fetches; - at most 3 concurrent requests per host;
- at most 500 pages fetched in a single site scan;
- at most 1 site scan and 30 single-page checks per customer workspace per hour;
- scheduled scans run at most once every 7 days per site;
- a 20 second total timeout per request;
- at most 5 redirects followed;
- reads at most 5 MB of a response body;
GETandHEADonly. It never submits a form, never signs in, and never attempts to reach anything behind authentication.
What it fetches, and what it does with it
AnsoarBot fetches publicly reachable pages on sites our customers have asked us to check: their own site, and in some cases a competitor's public pages for comparison. It reads robots.txt, sitemaps and page HTML.
Page content is used to produce the checks and recommendations shown to the customer whose workspace requested them. We do not sell page content, and we do not use it to train models of our own.
If you did not expect to see it
AnsoarBot fetches a site because an Ansoar customer entered that site in their workspace, which usually means it is their own. If you believe your site is being checked by someone with no authority to do so, email hello@ansoar.io with the domain and a sample log line and we will investigate.
Contact
Questions, crawl-rate concerns and complaints: hello@ansoar.io. If you need us to slow down or stop, say so and we will act on it. We would rather hear from you than be blocked.