Ensuring AI crawlers can access your website

For AI answers to fetch and cite your pages, the relevant crawlers need permission to visit your website in your robots.txt. This article shows where to see those permissions in KI-Readiness, how KI Score evaluates your robots.txt, and where the check reaches its limits.

Visibility crawlers and training crawlers

KI Score distinguishes between two groups of crawlers:

• Visibility crawlers fetch content for a search index or live for an AI answer, for example OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, Googlebot, and Bingbot. If you block one of them, the corresponding AI system can no longer fetch your page, so it can't cite it either.
• Training crawlers only collect training data, for example GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, and CCBot. Whether you make your content available for training is a strategic decision. Blocking these crawlers doesn't cost you any immediate visibility, and KI Score never counts it as an error.

In other words, you can shut out training crawlers and still stay visible in AI answers, as long as the visibility crawlers are allowed.

KI Score checks your robots.txt against its entire stored crawler catalog, including further AI crawlers such as Applebot, Amazonbot, or Bytespider. That way you see who else you're shutting out. These further crawlers don't count toward the visibility crawler check.

The “KI-Crawler” table

1. In the sidebar, open KI-Readiness under Optimierung (Optimization).
2. If no analysis exists yet, click Analyse starten (Start analysis). KI-Readiness doesn't use any credits.
3. Stay in the KI-Crawler (AI crawlers) section at the top, which is shown first. The KI-Crawler card's subtitle reads “Freigaben laut robots.txt. Sperren durch Firewall oder CDN (etwa Bot-Schutz) erkennt diese Prüfung nicht.” (Permissions according to robots.txt. This check doesn't detect blocks by a firewall or CDN, such as bot protection.)

At the top right, the card states how many of the checked crawlers are allowed. Use the Alle (All), Sichtbarkeit (Visibility), Training, Erlaubt (Allowed), and Blockiert (Blocked) buttons to filter the table. It has four columns:

A blocked visibility crawler that respects robots.txt according to its provider is highlighted as an error; a blocked training crawler is not.

The check covers six visibility crawlers that, according to their providers, respect robots.txt: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Claude-User, Googlebot, and Bingbot. ChatGPT-User and Perplexity-User appear in the table only as a hint, because according to their providers they barely respect robots.txt. Training crawlers never lead to an open item. If your robots.txt blocks one of the six crawlers, you'll find the item “Sichtbarkeits-Crawler freigeben” (Allow visibility crawlers) in the Nächste Schritte (Next steps) section.

How KI Score evaluates your robots.txt

KI Score reads your website's /robots.txt file and applies its rules to the address you entered, and, if that address redirects, to the redirect target as well. The following rules apply:

• If there's a group with the crawler's name, only that group applies. Otherwise, the User-agent: * group applies.
• The wildcards * (any characters) and $ (end of path) are supported.
• If several rules match, the longest match wins. If two matches are the same length, Allow wins.
• If robots.txt can't be reached because the server responds with a 5xx error or 429, or the request times out, this counts as a complete block, just as it does for real crawlers. All crawlers are then shown as Blockiert.
• If the file is missing (other 4xx responses), or the server returns an HTML page instead of a robots.txt, all crawlers count as allowed.

What the check doesn't detect

The table only shows what your robots.txt allows. KI Score never fetches your website using the crawlers' user agents; it uses its own identifier (“KiScore-Checker/2.0”). As a result, blocks that only take effect on your server, such as a firewall, a CDN, a web application firewall, or bot protection, remain invisible. A crawler marked Erlaubt can still be turned away in practice.

Check those settings directly with your hosting or CDN provider. You can see whether crawlers actually reach your website in Traffic Overview, provided you've set up the traffic integration.

Improving crawler permissions

1. Filter the KI-Crawler table with Blockiert and find every visibility crawler with this status.
2. Open your robots.txt and find the rule that blocks the crawler, either in its own group or under User-agent: *.
3. Allow the crawler, for example with a dedicated group User-agent: OAI-SearchBot and Allow: /.
4. Decide separately whether you want to allow or block the training crawlers.
5. Make sure /robots.txt is served reliably and without server errors.
6. In KI-Readiness, click Neu analysieren (Analyze again) to check the result.

Note: If a rule for User-agent: * blocks your entire website, it also affects Googlebot and Bingbot, unless they have their own group. KI Score maps these two crawlers to AI answers from Google and Microsoft Copilot.