Which AI crawlers visit your site most?
How to see in Traffic Overview which AI crawlers and providers fetch your website most often and which pages they visited most recently.
Last updated 23 days ago
Traffic Overview shows you which AI crawlers fetch your website most often, which providers they belong to, what they fetch your pages for, and which pages were fetched most recently. It only works if your website sends traffic data to KI Score from the server side. Without that integration, the page shows no analytics.
The top crawlers
To open the crawler view:
- In the sidebar, open KI-Traffic (AI traffic) → Traffic Overview.
- Under Zeitraum (time period) at the top, choose the period you want, such as Letzte 30 Tage (last 30 days), or mark your own range in the calendar.
- Switch to the KI-Crawler (AI crawlers) tab.
The Crawler table lists the ten crawlers detected most often in the selected period, with the columns Anzahl (count) and Anteil (share). The share refers to all requests in the period. Each crawler is identified by its user agent, such as GPTBot, ClaudeBot, or PerplexityBot. The Abrufe im Zeitverlauf (requests over time) chart shows how requests are spread across the period. The total appears at the top in the Abrufe durch KI-Crawler (requests by AI crawlers) metric, together with the number of different crawlers.
Grouped by provider
The Anbieter (providers) table groups the same requests by company, for example OpenAI, Anthropic, Google, Perplexity, Microsoft, Meta, Amazon, Apple, or Common Crawl. It shows at most the ten most frequent providers, each with its Anzahl and Anteil.
A provider bundles all of a company's crawlers, regardless of what they are used for. For example, OpenAI covers both GPTBot and OAI-SearchBot.
Visibility and training crawlers
Traffic Overview assigns every known crawler a purpose, using the same classification as KI-Readiness (AI readiness):
- Visibility crawlers fetch content for a search index or live for AI answers, such as OAI-SearchBot, PerplexityBot, or Googlebot. Blocking them costs you visibility in AI answers.
- Training crawlers collect training data, such as GPTBot, CCBot, or Bytespider. Blocking them is a strategic decision and usually doesn't cost immediate visibility.
The Sichtbarkeits-Crawler (visibility crawlers) and Trainings-Crawler (training crawlers) metrics and the Zweck der Abrufe (purpose of requests) card show how requests are split between the two groups. Requests without a known crawler name appear there as Ohne Zuordnung (unassigned). The Neueste Abrufe (latest requests) table shows the purpose for each individual request.
Which paths are fetched
To see which pages a crawler fetched, use the Neueste Abrufe table. It shows the most recently detected requests, with at most 25 rows. Each row contains the crawler with its provider and purpose, the Abgerufene Seite (fetched page), the IP-Adresse (IP address), and the Zeit (time) of the request.
Note: KI Score does not offer a breakdown by path. You won't see a ranking of the most frequently fetched pages, only the latest individual requests. Traffic Overview also has no file export.
Tip: KI-Readiness checks whether your robots.txt allows visibility and training crawlers. Traffic Overview only shows which crawlers actually fetched your website.
What teams say about KI Score
“We finally see week by week where our brand shows up in ChatGPT and Perplexity.”
“The score finally gave us one number the whole team actually understands.”
“Instead of guessing which prompts matter, we now have real data.”
“Within the first week we already saw which competitor was beating us in AI answers.”
“Setup took one afternoon, not weeks of onboarding.”
“Finally a tool that takes AI Overviews as seriously as classic SEO.”