Blomega Data Refinery / registry
Which sites actually block AI crawlers, measured from robots.txt
256 records, built 2026-09-18. Machine-readable: JSON and JSON-LD.
| Name | Sector | Measured | AI crawlers disallowed | Allows every AI crawler | Licensed to | TDM reservation | Grade | Source |
|---|---|---|---|---|---|---|---|---|
| 500px.com | stock_media | 2026-09-17 | not established | yes | not established | no | B | source |
| adobe.com | ai_company | 2026-09-17 | not established | yes | not established | not established | B | source |
| advancelocal.com | news_publisher | 2026-09-17 | not established | yes | not established | not established | B | source |
| afm.org | music | 2026-09-17 | not established | yes | not established | no | B | source |
| afp.com | news_publisher | 2026-09-17 | not established | yes | Mistral AI | not established | B | source |
| airbnb.com | ecommerce | 2026-09-17 | ImagesiftBot | no | not established | no | B | source |
| alamy.com | stock_media | 2026-09-17 | not established | yes | not established | no | B | source |
| alibaba.com | ecommerce | 2026-09-17 | GPTBot, Google-Extended | no | not established | no | B | source |
| aliexpress.com | ecommerce | 2026-09-17 | not established | yes | not established | no | B | source |
| aljazeera.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, Bytespider, PerplexityBot, cohere-ai | no | not established | no | B | source |
| amazon.com | ecommerce | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Google-Extended, CCBot, Bytespider, PerplexityBot, Perplexity-User, meta-externalagent, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| aninews.in | news_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| anthropic.com | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| apnews.com | news_publisher | 2026-09-17 | GPTBot, ClaudeBot, Claude-User, anthropic-ai, CCBot, Amazonbot, Applebot-Extended, PerplexityBot, cohere-ai, Timpibot | no | Google, Microsoft, OpenAI | no | B | source |
| apple.com | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| archive.org | ugc_platform | 2026-09-17 | not established | yes | not established | no | B | source |
| archiveofourown.org | ugc_platform | 2026-09-17 | GPTBot, ChatGPT-User, CCBot | no | not established | no | B | source |
| arstechnica.com | news_publisher | 2026-09-17 | not established | not established | not established | not established | C | source |
| artstation.com | stock_media | 2026-09-17 | not established | yes | not established | no | B | source |
| arxiv.org | academic_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| ascap.com | music | 2026-09-17 | not established | yes | not established | no | B | source |
| authorsguild.org | trade_body | 2026-09-17 | not established | yes | not established | no | B | source |
| automattic.com | ugc_platform | 2026-09-17 | not established | yes | OpenAI and Midjourney | no | B | source |
| axelspringer.com | news_publisher | 2026-09-17 | not established | yes | OpenAI | no | B | source |
| axios.com | news_publisher | 2026-09-17 | CCBot, Bytespider, Amazonbot, FacebookBot, Diffbot, ImagesiftBot | no | OpenAI | no | B | source |
| bandcamp.com | ugc_platform | 2026-09-17 | GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Amazonbot, meta-externalagent, FacebookBot, Diffbot, ImagesiftBot, omgili | no | not established | no | B | source |
| bbc.co.uk | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, cohere-ai, Diffbot, omgili | no | not established | no | B | source |
| behance.net | ugc_platform | 2026-09-17 | GPTBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, FacebookBot, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| believe.com | music | 2026-09-17 | not established | yes | Suno | not established | B | source |
| bestbuy.com | ecommerce | 2026-09-17 | not established | yes | not established | not established | B | source |
| bloomberg.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, meta-externalagent, cohere-ai, Diffbot | no | not established | no | B | source |
| bmg.com | music | 2026-09-17 | not established | yes | Suno | no | B | source |
| bmi.com | music | 2026-09-17 | not established | yes | not established | not established | B | source |
| booking.com | ecommerce | 2026-09-17 | not established | yes | not established | no | B | source |
| brave.com | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| bria.ai | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| brightdata.com | data_broker | 2026-09-17 | not established | yes | not established | no | B | source |
| businessinsider.com | news_publisher | 2026-09-17 | ClaudeBot, anthropic-ai, CCBot, Bytespider, Diffbot, Timpibot | no | OpenAI | no | B | source |
| cambridge.org | academic_publisher | 2026-09-17 | ChatGPT-User | no | not established | no | B | source |
| canada.ca | government | 2026-09-17 | not established | yes | not established | not established | B | source |
| canva.com | stock_media | 2026-09-17 | GPTBot, ClaudeBot, anthropic-ai, CCBot, Bytespider, Applebot-Extended, meta-externalagent, FacebookBot, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| causaly.com | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| cbsnews.com | news_publisher | 2026-09-17 | GPTBot | no | not established | no | B | source |
| cdc.gov | government | 2026-09-17 | not established | yes | not established | not established | B | source |
| census.gov | government | 2026-09-17 | not established | yes | not established | no | B | source |
| chicagotribune.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, Diffbot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | yes | B | source |
| clearview.ai | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| cnn.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| cohere.com | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| concord.com | music | 2026-09-17 | not established | yes | not established | no | B | source |
| condenast.com | news_publisher | 2026-09-17 | not established | yes | Amazon, OpenAI | not established | B | source |
| congress.gov | government | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| copyright.gov | government | 2026-09-17 | not established | yes | not established | not established | B | source |
| corriere.it | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, Timpibot | no | not established | no | B | source |
| cosmopolitan.com | news_publisher | 2026-09-17 | not established | yes | OpenAI | no | B | source |
| costco.com | ecommerce | 2026-09-17 | not established | yes | not established | no | B | source |
| dailymail.co.uk | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| data.gov | government | 2026-09-17 | not established | yes | not established | no | B | source |
| depositphotos.com | stock_media | 2026-09-17 | not established | yes | not established | no | B | source |
| deviantart.com | ugc_platform | 2026-09-17 | not established | yes | not established | no | B | source |
| discord.com | ugc_platform | 2026-09-17 | not established | yes | not established | no | B | source |
| disney.com | studio | 2026-09-17 | GPTBot, Google-Extended | no | OpenAI | no | B | source |
| dotdashmeredith.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, Timpibot | no | OpenAI | no | B | source |
| dowjones.com | news_publisher | 2026-09-17 | not established | yes | not established | not established | B | source |
| dreamstime.com | stock_media | 2026-09-17 | meta-externalagent | no | not established | no | B | source |
| dribbble.com | ugc_platform | 2026-09-17 | not established | yes | not established | no | B | source |
| dw.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| ebay.com | ecommerce | 2026-09-17 | GPTBot, ClaudeBot, anthropic-ai, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, meta-externalagent | no | not established | no | B | source |
| ec.europa.eu | government | 2026-09-17 | not established | yes | not established | no | B | source |
| economist.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Applebot-Extended, PerplexityBot, Perplexity-User | no | not established | no | B | source |
| elevenlabs.io | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| elpais.com | news_publisher | 2026-09-17 | ClaudeBot, Claude-User, CCBot, Bytespider, Amazonbot, PerplexityBot, Perplexity-User | no | OpenAI | no | B | source |
| elsevier.com | academic_publisher | 2026-09-17 | not established | yes | not established | yes | B | source |
| engadget.com | news_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| envato.com | stock_media | 2026-09-17 | CCBot | no | Bria AI | no | B | source |
| esquire.com | news_publisher | 2026-09-17 | not established | yes | OpenAI | no | B | source |
| etsy.com | ecommerce | 2026-09-17 | not established | yes | not established | no | B | source |
| expedia.com | ecommerce | 2026-09-17 | not established | yes | not established | no | B | source |
| facebook.com | ugc_platform | 2026-09-17 | ChatGPT-User, OAI-SearchBot, Claude-User, anthropic-ai, CCBot, Bytespider, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| fandom.com | ugc_platform | 2026-09-17 | not established | not established | not established | not established | C | source |
| faz.net | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, meta-externalagent, cohere-ai, Diffbot, omgili, Timpibot, AI2Bot | no | not established | no | B | source |
| flickr.com | ugc_platform | 2026-09-17 | GPTBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| forbes.com | news_publisher | 2026-09-17 | GPTBot, ClaudeBot, anthropic-ai, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili | no | not established | no | B | source |
| foxnews.com | news_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| freepik.com | stock_media | 2026-09-17 | not established | yes | not established | no | B | source |
| ft.com | news_publisher | 2026-09-17 | ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, omgili | no | OpenAI | no | B | source |
| futureplc.com | news_publisher | 2026-09-17 | Bytespider, Amazonbot, meta-externalagent, Diffbot, omgili, AI2Bot | no | OpenAI | no | B | source |
| gema.de | music | 2026-09-17 | not established | yes | not established | no | B | source |
| gettyimages.com | stock_media | 2026-09-17 | not established | yes | Bria AI, OpenAI, Perplexity AI, Picsart | no | B | source |
| github.com | ugc_platform | 2026-09-17 | Bytespider | no | not established | no | B | source |
| gitlab.com | ugc_platform | 2026-09-17 | not established | yes | not established | no | B | source |
| global.oup.com | academic_publisher | 2026-09-17 | not established | not established | not established | not established | C | source |
| goodreads.com | ugc_platform | 2026-09-17 | GPTBot, CCBot | no | not established | no | B | source |
| google.com | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| gov.uk | government | 2026-09-17 | not established | yes | not established | no | B | source |
| gutenberg.org | ugc_platform | 2026-09-17 | not established | yes | not established | no | B | source |
| hachettebookgroup.com | book_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| harpercollins.com | book_publisher | 2026-09-17 | not established | yes | Microsoft | no | B | source |
| hearst.com | news_publisher | 2026-09-17 | not established | not established | Amazon, OpenAI | not established | C | source |
| homedepot.com | ecommerce | 2026-09-17 | not established | not established | not established | not established | C | source |
| huffpost.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, Timpibot | no | not established | no | B | source |
| huggingface.co | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| ifpi.org | trade_body | 2026-09-17 | not established | yes | not established | no | B | source |
| ikea.com | ecommerce | 2026-09-17 | not established | yes | not established | no | B | source |
| imdb.com | ugc_platform | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| imgur.com | ugc_platform | 2026-09-17 | not established | yes | not established | no | B | source |
| independent.co.uk | news_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| instagram.com | ugc_platform | 2026-09-17 | not established | yes | not established | not established | B | source |
| investopedia.com | news_publisher | 2026-09-17 | ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, ImagesiftBot, omgili, Timpibot, Webzio-Extended | no | OpenAI | no | B | source |
| ioppublishing.org | academic_publisher | 2026-09-17 | not established | yes | Cashmere | not established | B | source |
| irs.gov | government | 2026-09-17 | not established | yes | not established | no | B | source |
| istockphoto.com | stock_media | 2026-09-17 | not established | yes | Bria AI, OpenAI, Perplexity AI, Picsart | no | B | source |
| jamendo.com | music | 2026-09-17 | not established | yes | not established | no | B | source |
| japantimes.co.jp | news_publisher | 2026-09-17 | not established | not established | not established | not established | C | source |
| jstor.org | academic_publisher | 2026-09-17 | GPTBot, ClaudeBot, Google-Extended, CCBot | no | not established | no | B | source |
| kaggle.com | ugc_platform | 2026-09-17 | not established | yes | not established | not established | B | source |
| kobaltmusic.com | music | 2026-09-17 | not established | yes | ElevenLabs, Spotify, Udio | no | B | source |
| latimes.com | news_publisher | 2026-09-17 | GPTBot, ClaudeBot, Bytespider, Diffbot, ImagesiftBot, omgili | no | not established | no | B | source |
| lefigaro.fr | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Amazonbot, cohere-ai, Diffbot, omgili, Webzio-Extended | no | not established | yes | B | source |
| lemonde.fr | news_publisher | 2026-09-17 | ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, FacebookBot, cohere-ai, Diffbot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | OpenAI | no | B | source |
| lgresearch.ai | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| linkedin.com | ugc_platform | 2026-09-17 | not established | yes | not established | not established | B | source |
| lionsgate.com | studio | 2026-09-17 | not established | yes | Runway | no | B | source |
| lovo.ai | ai_company | 2026-09-17 | not established | not established | not established | not established | C | source |
| macmillan.com | book_publisher | 2026-09-17 | not established | not established | not established | not established | C | source |
| medium.com | ugc_platform | 2026-09-17 | GPTBot, ClaudeBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, FacebookBot | no | not established | no | B | source |
| merlinnetwork.org | music | 2026-09-17 | not established | yes | ElevenLabs, Spotify, Udio | no | B | source |
| meta.com | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| microsoft.com | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| midjourney.com | ai_company | 2026-09-17 | GPTBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili | no | not established | not established | B | source |
| mistral.ai | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| mitpress.mit.edu | academic_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | not established | B | source |
| moonvalley.com | ai_company | 2026-09-17 | not established | yes | not established | not established | B | source |
| nasa.gov | government | 2026-09-17 | not established | yes | not established | no | B | source |
| nature.com | academic_publisher | 2026-09-17 | GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili | no | not established | yes | B | source |
| nbcnews.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, omgili | no | not established | no | B | source |
| news.ycombinator.com | ugc_platform | 2026-09-17 | not established | yes | not established | no | B | source |
| newscorp.com | news_publisher | 2026-09-17 | not established | yes | Meta, OpenAI | no | B | source |
| newsday.com | news_publisher | 2026-09-17 | GPTBot, CCBot | no | not established | no | B | source |
| newsmax.com | news_publisher | 2026-09-17 | GPTBot | no | Meta | not established | B | source |
| newyorker.com | news_publisher | 2026-09-17 | ClaudeBot, Claude-User, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, cohere-ai, Diffbot, Timpibot, Webzio-Extended | no | OpenAI | no | B | source |
| nih.gov | government | 2026-09-17 | not established | yes | not established | not established | B | source |
| nike.com | ecommerce | 2026-09-17 | not established | yes | not established | no | B | source |
| nineentertainment.com.au | news_publisher | 2026-09-17 | not established | yes | Microsoft | not established | B | source |
| nmpa.org | music | 2026-09-17 | not established | yes | KLAY Vision, Udio | no | B | source |
| npr.org | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Applebot-Extended, PerplexityBot, FacebookBot, cohere-ai, Diffbot, omgili | no | not established | no | B | source |
| nvidia.com | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| nypost.com | news_publisher | 2026-09-17 | not established | not established | Meta, OpenAI | not established | C | source |
| nytimes.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot | no | Amazon | no | B | source |
| openai.com | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| openlibrary.org | ugc_platform | 2026-09-17 | not established | yes | not established | no | B | source |
| oxylabs.io | data_broker | 2026-09-17 | not established | yes | not established | no | B | source |
| patreon.com | ugc_platform | 2026-09-17 | PerplexityBot, FacebookBot, Timpibot | no | not established | no | B | source |
| pearson.com | book_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| penguinrandomhouse.com | book_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| people.com | news_publisher | 2026-09-17 | ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, ImagesiftBot, omgili, Timpibot, Webzio-Extended | no | OpenAI | no | B | source |
| perfil.com | news_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| perplexity.ai | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| pexels.com | stock_media | 2026-09-17 | GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, PerplexityBot | no | not established | no | B | source |
| photobucket.com | ugc_platform | 2026-09-17 | not established | yes | not established | no | B | source |
| picsart.com | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| pinterest.com | ugc_platform | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | not established | B | source |
| pixabay.com | stock_media | 2026-09-17 | GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended | no | not established | no | B | source |
| politico.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | OpenAI | not established | B | source |
| pond5.com | stock_media | 2026-09-17 | not established | not established | not established | not established | C | source |
| press.princeton.edu | academic_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | Cashmere | no | B | source |
| prisa.com | news_publisher | 2026-09-17 | not established | yes | OpenAI | not established | B | source |
| publishers.org | trade_body | 2026-09-17 | not established | yes | not established | no | B | source |
| pubmed.ncbi.nlm.nih.gov | academic_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| quora.com | ugc_platform | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, CCBot, Bytespider, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | not established | B | source |
| rawstory.com | news_publisher | 2026-09-17 | GPTBot, Google-Extended, CCBot, PerplexityBot | no | not established | not established | B | source |
| reddit.com | ugc_platform | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | Google, OpenAI | no | B | source |
| reka.ai | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| reuters.com | news_publisher | 2026-09-17 | GPTBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | Meta | no | B | source |
| riaa.com | trade_body | 2026-09-17 | not established | yes | not established | no | B | source |
| rossintelligence.com | ai_company | 2026-09-17 | not established | yes | not established | not established | B | source |
| roundhillmusic.com | music | 2026-09-17 | not established | yes | not established | not established | B | source |
| runwayml.com | ai_company | 2026-09-17 | not established | yes | not established | not established | B | source |
| saatchiart.com | stock_media | 2026-09-17 | Bytespider | no | not established | not established | B | source |
| sagepub.com | academic_publisher | 2026-09-17 | CCBot, Bytespider, Amazonbot, meta-externalagent, cohere-ai, Diffbot, ImagesiftBot, omgili | no | Causaly | yes | B | source |
| samsung.com | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| schibsted.com | news_publisher | 2026-09-17 | not established | yes | OpenAI | no | B | source |
| scholastic.com | book_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| science.org | academic_publisher | 2026-09-17 | GPTBot, ChatGPT-User, anthropic-ai, Google-Extended, CCBot, Applebot-Extended | no | not established | yes | B | source |
| sciencedirect.com | academic_publisher | 2026-09-17 | GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | yes | B | source |
| scmp.com | news_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| seattletimes.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| sec.gov | government | 2026-09-17 | not established | yes | not established | not established | B | source |
| serpapi.com | data_broker | 2026-09-17 | not established | yes | not established | no | B | source |
| shopify.com | ecommerce | 2026-09-17 | not established | yes | not established | no | B | source |
| shutterstock.com | stock_media | 2026-09-17 | GPTBot, ClaudeBot, CCBot, meta-externalagent | no | Amazon, Apple, Google, LG AI Research, Meta, NVIDIA, OpenAI, Reka AI | no | B | source |
| simonandschuster.com | book_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| slate.com | news_publisher | 2026-09-17 | GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, FacebookBot, Diffbot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| smh.com.au | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, Webzio-Extended | no | Microsoft | not established | B | source |
| socan.ca | music | 2026-09-17 | not established | yes | not established | no | B | source |
| sonymusic.com | music | 2026-09-17 | not established | yes | not established | no | B | source |
| soundcloud.com | ugc_platform | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, meta-externalagent, cohere-ai | no | not established | no | B | source |
| spiegel.de | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| spotify.com | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| springer.com | academic_publisher | 2026-09-17 | not established | yes | not established | yes | B | source |
| stability.ai | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| stackexchange.com | ugc_platform | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | Google, OpenAI | not established | B | source |
| stackoverflow.com | ugc_platform | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | Google, OpenAI | not established | B | source |
| stock.adobe.com | stock_media | 2026-09-17 | GPTBot, anthropic-ai, Google-Extended, CCBot, Applebot-Extended | no | Adobe | no | B | source |
| storyblocks.com | stock_media | 2026-09-17 | GPTBot, ClaudeBot, CCBot, Bytespider, Applebot-Extended, Diffbot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| substack.com | ugc_platform | 2026-09-17 | not established | yes | not established | no | B | source |
| suno.com | ai_company | 2026-09-17 | not established | yes | not established | no | B | source |
| tampabay.com | news_publisher | 2026-09-17 | GPTBot, ClaudeBot, anthropic-ai, CCBot, Bytespider, Diffbot | no | not established | no | B | source |
| target.com | ecommerce | 2026-09-17 | not established | yes | not established | no | B | source |
| taylorandfrancis.com | academic_publisher | 2026-09-17 | not established | yes | Microsoft | no | B | source |
| techcrunch.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Applebot-Extended, FacebookBot, cohere-ai, Diffbot, omgili | no | not established | no | B | source |
| techradar.com | news_publisher | 2026-09-17 | Bytespider, Amazonbot, meta-externalagent, Diffbot, omgili, AI2Bot | no | OpenAI | no | B | source |
| telegraph.co.uk | news_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| theatlantic.com | news_publisher | 2026-09-17 | ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, cohere-ai, Diffbot, omgili, Timpibot | no | OpenAI | no | B | source |
| theglobeandmail.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, PerplexityBot, Perplexity-User, meta-externalagent, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| theguardian.com | news_publisher | 2026-09-17 | ClaudeBot, Claude-User, anthropic-ai, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, meta-externalagent, FacebookBot | no | OpenAI | no | B | source |
| thehill.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Applebot-Extended, PerplexityBot, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili | no | not established | no | B | source |
| thehindu.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, CCBot, Bytespider, Amazonbot, PerplexityBot, meta-externalagent, FacebookBot | no | not established | no | B | source |
| theintercept.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, anthropic-ai, Google-Extended, CCBot, Amazonbot, PerplexityBot | no | not established | no | B | source |
| thetimes.co.uk | news_publisher | 2026-09-17 | ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | Meta, OpenAI | no | B | source |
| theverge.com | news_publisher | 2026-09-17 | ChatGPT-User, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | OpenAI | no | B | source |
| thewaltdisneycompany.com | studio | 2026-09-17 | not established | yes | OpenAI | no | B | source |
| thomsonreuters.com | news_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| tiktok.com | ugc_platform | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, AI2Bot | no | not established | no | B | source |
| time.com | news_publisher | 2026-09-17 | not established | yes | OpenAI | no | B | source |
| tomshardware.com | news_publisher | 2026-09-17 | Bytespider, Amazonbot, meta-externalagent, Diffbot, omgili, AI2Bot | no | OpenAI | no | B | source |
| tripadvisor.com | ugc_platform | 2026-09-17 | GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, cohere-ai | no | not established | no | B | source |
| tumblr.com | ugc_platform | 2026-09-17 | not established | not established | OpenAI and Midjourney | not established | C | source |
| twitch.tv | ugc_platform | 2026-09-17 | not established | yes | not established | no | B | source |
| udio.com | ai_company | 2026-09-17 | not established | yes | not established | not established | B | source |
| un.org | government | 2026-09-17 | not established | yes | not established | not established | B | source |
| universalmusic.com | music | 2026-09-17 | GPTBot | no | ElevenLabs, Spotify, Udio | no | B | source |
| unsplash.com | stock_media | 2026-09-17 | ClaudeBot, Bytespider, ImagesiftBot | no | not established | no | B | source |
| usa.gov | government | 2026-09-17 | not established | yes | not established | no | B | source |
| usatoday.com | news_publisher | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| uspto.gov | government | 2026-09-17 | not established | yes | not established | no | B | source |
| vg.no | news_publisher | 2026-09-17 | ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | OpenAI | no | B | source |
| vimeo.com | ugc_platform | 2026-09-17 | GPTBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, FacebookBot, Diffbot, ImagesiftBot, omgili | no | not established | no | B | source |
| vogue.com | news_publisher | 2026-09-17 | ClaudeBot, Claude-User, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, cohere-ai, Diffbot, Timpibot, Webzio-Extended | no | OpenAI | no | B | source |
| vox.com | news_publisher | 2026-09-17 | ChatGPT-User, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | OpenAI | no | B | source |
| voxmedia.com | news_publisher | 2026-09-17 | not established | yes | OpenAI | no | B | source |
| walmart.com | ecommerce | 2026-09-17 | not established | yes | not established | no | B | source |
| washingtonpost.com | news_publisher | 2026-09-17 | ClaudeBot, anthropic-ai, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, meta-externalagent, FacebookBot, Diffbot, ImagesiftBot, omgili | no | OpenAI | no | B | source |
| whitehouse.gov | government | 2026-09-17 | not established | yes | not established | no | B | source |
| who.int | government | 2026-09-17 | CCBot | no | not established | no | B | source |
| wikihow.com | ugc_platform | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| wikipedia.org | ugc_platform | 2026-09-17 | not established | yes | not established | no | B | source |
| wiley.com | academic_publisher | 2026-09-17 | not established | yes | not established | no | B | source |
| wired.com | news_publisher | 2026-09-17 | ClaudeBot, Claude-User, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, cohere-ai, Diffbot, Timpibot, Webzio-Extended | no | OpenAI | no | B | source |
| wmg.com | music | 2026-09-17 | not established | yes | Suno, Udio | no | B | source |
| wordpress.com | ugc_platform | 2026-09-17 | not established | yes | OpenAI and Midjourney | no | B | source |
| worldbank.org | government | 2026-09-17 | not established | yes | not established | no | B | source |
| wsj.com | news_publisher | 2026-09-17 | ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | Meta, OpenAI | no | B | source |
| x.com | ugc_platform | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| yelp.com | ugc_platform | 2026-09-17 | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, PerplexityBot, Perplexity-User, meta-externalagent, FacebookBot, cohere-ai, Diffbot, ImagesiftBot, omgili, Timpibot, AI2Bot, Webzio-Extended | no | not established | no | B | source |
| youtube.com | ugc_platform | 2026-09-17 | not established | yes | 18 named third-party AI companies | no | B | source |
Cite as: Blomega Data Refinery, https://data.blomega.com, CC BY 4.0. Published by
Blomega (Wikidata Q141048865). Unknown values are shown as
not established, never guessed.