Free tool
AI Crawler Simulator
The first requirement for entering AI answers: bots must be able to crawl your site. Enter a URL and see which AI bots can reach it.
How the crawler simulator works
- Two separate tests run on the URL you enter. First we fetch the site's robots.txt and evaluate it bot by bot for that exact path — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider and more. The longest matching rule wins, the same logic real crawlers apply.
- Then comes the live probe: we send real GET requests to the page with the user agents of GPTBot, ClaudeBot and PerplexityBot and show the HTTP status each one receives. This catches pages that look allowed in robots.txt but where the server or a security layer returns 403 to bot user agents.
Two limits worth knowing: robots.txt is a declaration, and some bots — training crawlers especially — do not always honor it. The probe also originates from our server, so a CDN that identifies bots by IP address may treat the real crawler differently than it treats us.
How to read the crawler simulation
The picture you want: answer bots (OAI-SearchBot, PerplexityBot, ChatGPT-User) allowed, and the probe returning 200. A blocked answer bot means that page simply cannot appear in AI search — fix that first. Blocking training bots like GPTBot or CCBot can be a deliberate choice; if you blocked them without meaning to, revisit the decision. Allowed in robots but 403/503 in the probe points away from robots.txt and toward your server or WAF. For a line-by-line take on which bots to allow and which to block: robots.txt for AI crawlers
AI answers change every day. Stay in the know.
If you're not showing up today, the customer goes to your competitor; if you are, you need to hold that position. Either way, the first step is the same: start measuring.
No credit card · 3-minute setup · cancel anytime