Understanding AI crawler traffic
What the “KI-Crawler” tab in Traffic Overview shows, how KI Score detects crawlers, and where detection reaches its limits.
Last updated 23 days ago
This article explains what you see in the KI-Crawler (AI crawlers) tab of Traffic Overview: which AI crawlers fetch your website, how often, for what purpose, and which pages. You'll also learn how KI Score recognizes a crawler and where that recognition has limits.
What counts as a crawler request
A crawler request is a single request to your website that KI Score attributes to a known AI crawler. This requires the server-side integration: your server reports requests to KI Score, and KI Score checks the user agent sent along with each one. If it matches one of the 33 stored bot names, such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Googlebot, or Bingbot, the request is saved. Requests without a matching user agent don't count as crawler requests.
Until the integration is set up, this area stays empty. You'll find the setup in the sidebar under Einstellungen → Traffic-Tracking (Settings → Traffic tracking).
To open the crawler data:
- In the sidebar, open Traffic Overview in the “KI-Traffic” (AI traffic) group.
- Under Zeitraum (Time period) at the top, choose a preset such as Letzte 7 Tage, Letzte 14 Tage, or Letzte 30 Tage (last 7, 14, or 30 days), or mark your own range in the calendar. Then click Übernehmen (Apply). The default is the last 30 days.
- Switch from the KI-Verweise (AI referrals) tab, which is shown first, to the KI-Crawler tab.
The metrics at a glance
At the top of the KI-Crawler tab, three metrics cover the selected time period:
| Metric | Meaning |
|---|---|
| Abrufe durch KI-Crawler (Requests by AI crawlers) | The number of detected crawler requests. Below it, you see how many different crawlers were involved. |
| Sichtbarkeits-Crawler (Visibility crawlers) | Requests from crawlers that fetch content for AI answers, such as OAI-SearchBot or PerplexityBot, with their share of all requests |
| Trainings-Crawler (Training crawlers) | Requests from crawlers that collect training data, such as GPTBot or CCBot, with their share of all requests |
The metrics for human visits from AI answers are in the KI-Verweise tab.
Requests over time and their purpose
The Abrufe im Zeitverlauf (Requests over time) chart shows the number of requests from all AI crawlers combined, per day. Days are grouped in German time (Europe/Berlin).
The Zweck der Abrufe (Purpose of requests) card splits requests into Sichtbarkeit (Visibility) and Training, each with its count and share. Requests that KI Score can't assign to a purpose, for example because the crawler name is missing, appear there as Ohne Zuordnung (Unassigned).
Crawlers and providers
The Crawler table lists the ten most frequent crawler names with their Anzahl (count) and Anteil (share) of all requests in the period. The Anbieter (Providers) table groups requests by company, such as OpenAI, Anthropic, Google, or Perplexity, also showing the ten most frequent entries.
Latest requests
The Neueste Abrufe (Latest requests) table shows up to the 25 most recent requests in the selected period, with these columns:
- Crawler: the detected bot name, with its provider and purpose below it.
- Abgerufene Seite (Fetched page): the page the crawler fetched.
- IP-Adresse (IP address): the crawler's reported IP address, if available.
- Zeit (Time): the date and time of the request.
This lets you quickly see which pages AI crawlers fetched most recently.
Limits of detection
- Only the user agent counts. KI Score doesn't verify IP addresses. If another bot poses as GPTBot with a spoofed user agent, it's still counted as GPTBot.
- Only known names. Crawlers whose names aren't stored don't appear.
- Only reported requests. KI Score sees only the requests your integration reports.
- No read-only workspace. While your workspace is read-only, KI Score doesn't accept crawler requests.
- Observe, not control. KI Score doesn't block bots. Which crawlers your robots.txt allows is checked by KI-Readiness (AI readiness).
What teams say about KI Score
“We finally see week by week where our brand shows up in ChatGPT and Perplexity.”
“The score finally gave us one number the whole team actually understands.”
“Instead of guessing which prompts matter, we now have real data.”
“Within the first week we already saw which competitor was beating us in AI answers.”
“Setup took one afternoon, not weeks of onboarding.”
“Finally a tool that takes AI Overviews as seriously as classic SEO.”