AI crawlers: who runs them, what for, and whether robots.txt stops them

Name: AI crawlers: who runs them, what for, and whether robots.txt stops them
Creator: Blomega
License: https://creativecommons.org/licenses/by/4.0/

34 records, built 2026-09-18. Machine-readable: JSON and JSON-LD.

Name	Operator	Purpose	Operator says it honours robots.txt	Blocking it keeps you in search	Sites blocking it (of 245 measured)	Grade	Source
AI2Bot	Allen Institute for AI	training	not established	not established	41	C	source
Amazonbot	Amazon	mixed	yes	no	61	B	source
Amzn-SearchBot	Amazon	search_index	yes	yes	not established	B	source
Amzn-User	Amazon	user_fetch	no	not established	not established	B	source
anthropic-ai	Anthropic	not established	not established	not established	68	C	source
Applebot	Apple	mixed	yes	yes	not established	B	source
Applebot-Extended	Apple	training	yes	no	67	B	source
Bytespider	ByteDance	not established	not established	not established	84	C	source
CCBot	Common Crawl	mixed	yes	not established	88	B	source
ChatGPT-User	OpenAI	user_fetch	no	no	49	B	source
Claude-SearchBot	Anthropic	search_index	yes	yes	not established	B	source
Claude-User	Anthropic	user_fetch	yes	not established	43	B	source
ClaudeBot	Anthropic	training	yes	not established	79	B	source
cohere-ai	Cohere	not established	not established	not established	58	C	source
cohere-training-data-crawler	Cohere	not established	not established	not established	not established	C	source
Diffbot	Diffbot	not established	yes	not established	67	C	source
DuckAssistBot	DuckDuckGo	user_fetch	yes	no	not established	B	source
FacebookBot	Meta	not established	not established	not established	54	C	source
Google-CloudVertexBot	Google	user_fetch	yes	no	not established	B	source
Google-Extended	Google	mixed	yes	no	67	B	source
GoogleOther	Google	mixed	yes	no	not established	B	source
GPTBot	OpenAI	training	yes	no	74	B	source
ImagesiftBot	ImageSift	search_index	yes	not established	49	B	source
meta-externalagent	Meta	mixed	not established	not established	68	C	source
Meta-ExternalFetcher	Meta	user_fetch	no	not established	not established	B	source
MistralAI-Index	Mistral AI	search_index	not established	yes	not established	B	source
MistralAI-Training	Mistral AI	training	yes	no	not established	B	source
MistralAI-User	Mistral AI	user_fetch	not established	not established	not established	B	source
OAI-SearchBot	OpenAI	search_index	yes	yes	38	B	source
omgili	Webz.io	mixed	yes	not established	60	B	source
Perplexity-User	Perplexity AI	user_fetch	no	not established	41	B	source
PerplexityBot	Perplexity AI	search_index	yes	yes	61	B	source
Timpibot	Timpi	not established	not established	not established	51	C	source
Webzio-Extended	Webz.io	training	yes	not established	43	B	source

Cite as: Blomega Data Refinery, https://data.blomega.com, CC BY 4.0. Published by Blomega (Wikidata Q141048865). Unknown values are shown as not established, never guessed.