Which AI bot is requesting your website?
Vincent Sünderhauf
GPTBot appears in a server log. That does not establish that the website appears in a search answer. The name identifies a particular purpose. Treating every AI bot alike can lead to the wrong access decision.
Separate search from training
OpenAI distinguishes OAI-SearchBot for search from GPTBot for potential training data. ChatGPT-User retrieves pages in connection with user actions. Anthropic also documents separate bots for training, search and user requests. Use the provider’s documentation when setting rules; old bot lists can become inaccurate.
Allowing search access therefore need not mean allowing training. Blocking a training crawler does not automatically switch off every product from that provider either.
What does the request receive?
Follow an important page request through to the response. Does it receive the content or an access restriction? Are key facts missing without additional JavaScript? Are there redirects, errors or bot challenges?
A browser screenshot answers only some of those questions. Server logs and the response actually delivered help narrow the problem down. A user-agent name alone also does not verify a bot’s identity; use the verification methods or IP lists published by its operator for that purpose.
Record the intended access rules and check them again after a change. A successful request demonstrates technical access. Whether the page is subsequently cited is a separate observation.
More from the magazine
What teams say about KI Score
“We finally see week by week where our brand shows up in ChatGPT and Perplexity.”
“The score finally gave us one number the whole team actually understands.”
“Instead of guessing which prompts matter, we now have real data.”
“Within the first week we already saw which competitor was beating us in AI answers.”
“Setup took one afternoon, not weeks of onboarding.”
“Finally a tool that takes AI Overviews as seriously as classic SEO.”
Measured where your customers ask
Move your brand ahead in ChatGPT and co.
Set up in minutes, your first measurement included.