Almost every week, you read that “more and more websites are shutting out AI crawlers”. There are figures for the US, but hardly any for Germany. In each monthly crawl, the HTTP Archive records the robots.txt of millions of home pages and counts the Allow and Disallow lines for each identifier; a schema change in February 2026 made comparisons between months possible. This makes it possible to answer the question for Germany, over time and by size. In the AI readability study, I had already read the robots.txt of 100 large German shops individually; they serve as a cross-check here. The overview of my studies places this measurement alongside the others.
4.6 %
of German home pages name GPTBot
457,225 .de home pages with robots.txt rules, October 2026 crawl; worldwide 8.3 %
28.8 %
of German websites with a rank up to 100,000
1,088 home pages with a CrUX rank up to 100,000; worldwide excluding Germany 13.9 %
3.5 → 4.4 %
GPTBot named on the same German pages
432,997 home pages in both datasets, of which 352,715 and 360,897 respectively have rule groups; Disallow without Allow 2.4 → 2.6 %
4 of 77
large German shops fully block GPTBot
robots.txt from October 11, 2026; 11 name GPTBot, 3 explicitly allow it
HTTP Archive, monthly datasets February to October 2026, mobile, home pages; 100 large German shops, robots.txt from October 11, 2026.
Data and method
The basis is the HTTP Archive, a public dataset in Google BigQuery that loads the home pages of all websites with enough navigations in the Chrome User Experience Report (CrUX) each month. It also fetches robots.txt and counts the lines containing Allow, Disallow and Crawl-delay for each identifier (User-Agent). Since the schema change in February 2026 (metric in the source code), these counts have been stored in a form that allows comparisons between months; Google presented the analysis in April 2026 on the Search Off the Record podcast. I analyzed the monthly datasets from February to October 2026, using the mobile version and only home pages; each home page is measured once per monthly crawl, and the dataset date is not the fetch date (HTTP Archive FAQ).
- Germany covers all home pages whose host ends in
.de: 552,750 pages in October, of which 457,225 have a robots.txt containing at least one rule group. All percentages refer to these pages. Worldwide covers all 15,366,242 home pages in the dataset, 12,023,939 with rule groups. - Shops are home pages on which the HTTP Archive detected a shop technology (E-Commerce category, for example Shopware, Shopify, WooCommerce, JTL, Magento or generic cart detection): 96,915 in Germany and 3,533,514 worldwide with rule groups. If the HTTP Archive detects several shop technologies, I use one of them.
- Named means that the identifier is explicitly listed as a User-Agent, even if it shares a group with other identifiers or with
*. Disallow without Allow (block indicator) means that the entry for this identifier contains at least one Disallow and no Allow. The metric does not record paths, so it cannot distinguish a full block (Disallow: /) from a partial block. - Calibration: In the complete robots.txt files of the 100 large shops (77 with a readable file), I classified every entry for 21 AI identifiers: 119 entries in 17 shops. Of 62 entries that the indicator counts as a block (from twelve shops), 54 were full blocks and 8 were partial blocks. I have not checked this for other types of website.
- Size is the website’s CrUX popularity bucket (methodology): a rough measure based on the number of navigations in Chrome, not revenue. The raw buckets (1,000, 5,000, 10,000, 50,000, 100,000, 500,000, 1 million, 5 million, 10 million, 50 million) are shown cumulatively.
Rules in robots.txt follow the group principle: an identifier with a matching group follows that group; otherwise it follows the rules for *. Not naming GPTBot therefore does not necessarily mean allowing everything; it then follows the rules for all bots, which may also be a full block. The study counts only explicit naming and entries with Disallow without Allow.
Germany compared with worldwide
In October 2026, 4.6 % of German home pages name GPTBot in robots.txt, and 2.5 % have Disallow without Allow for it. Worldwide, the figures are 8.3 % and 3.5 %. Germany’s overall figures are therefore below the worldwide figures for each of the 13 identifiers and both measures. The order of bots is almost the same in both groups: GPTBot first, then ClaudeBot, Bytespider and CCBot; the search and live-fetch identifiers OAI-SearchBot, ChatGPT-User and Claude-Web are named least often.
- GPTBot 4.6 % (2.5 %)
- ClaudeBot 3.7 % (1.4 %)
- Bytespider 3.0 % (2.1 %)
- CCBot 2.9 % (1.6 %)
- Google-Extended 2.8 % (1.1 %)
- meta-externalagent 2.8 % (1.8 %)
- Amazonbot 2.6 % (1.4 %)
- PerplexityBot 2.3 % (0.7 %)
- ChatGPT-User 2.1 % (0.8 %)
- anthropic-ai 2.1 % (0.9 %)
- Applebot-Extended 2.1 % (0.9 %)
- OAI-SearchBot 1.7 % (0.5 %)
- Claude-Web 1.0 % (0.6 %)
HTTP Archive, October 2026 monthly dataset (date = 2026-10-01), mobile, home pages with robots.txt rule groups: 457,225 home pages with a host ending in .de.
- GPTBot 4.6 % → 8.3 % · n = 457,225
- ClaudeBot 3.7 % → 7.4 % · n = 457,225
- Bytespider 3.0 % → 5.5 % · n = 457,225
- CCBot 2.9 % → 6.3 % · n = 457,225
- Google-Extended 2.8 % → 6.3 % · n = 457,225
- meta-externalagent 2.8 % → 5.1 % · n = 457,225
- Amazonbot 2.6 % → 5.5 % · n = 457,225
- PerplexityBot 2.3 % → 4.4 % · n = 457,225
- ChatGPT-User 2.1 % → 3.9 % · n = 457,225
- OAI-SearchBot 1.7 % → 3.4 % · n = 457,225
HTTP Archive, October 2026 monthly dataset (date = 2026-10-01), mobile, home pages with robots.txt rule groups: 457,225 German home pages and 12,023,939 home pages worldwide.
For shops, the gap in naming rates is similar: 6.9 % of German shops name GPTBot (2.8 % with Disallow without Allow), compared with 11.7 % worldwide (2.7 %). What stands out is that shops worldwide name GPTBot more often than websites overall, yet the block indicator is no more frequent. Some of the naming may come from platform templates containing Allow; the counts do not show whether that is the reason.
Large websites block GPTBot more often
Sorting by size reverses the picture. Among German home pages with a CrUX rank up to 100,000, 28.8 % name GPTBot and 23.0 % have Disallow without Allow. Worldwide excluding Germany, the figures in the same bucket are 13.9 % and 10.2 %. Up to rank 10,000, Germany is at 48.5 % (132 pages), compared with 18.1 %. Up to rank 1 million, Germany is still slightly higher (10.7 % compared with 10.3 %); in the bucket up to 10 million, it is lower (5.0 % compared with 8.6 %).
- up to 10,000 (132 German pages) 48.5 % → 18.1 % · n = 132
- up to 100,000 (1,088 German pages) 28.8 % → 13.9 % · n = 1,088
- up to 1 million (13,455 German pages) 10.7 % → 10.3 % · n = 13,455
- up to 10 million (236,450 German pages) 5.0 % → 8.6 % · n = 236,450
- all (457,225 German pages) 4.6 % → 8.4 % · n = 457,225
HTTP Archive, October 2026 monthly dataset (date = 2026-10-01), mobile, home pages with robots.txt rule groups. Rank = the website’s CrUX popularity bucket. The bucket up to 1,000 contains only 15 German home pages and is included in “up to 10,000”.
In this sample, large German websites therefore name and block GPTBot more often than large websites in other countries, while Germany’s long tail has fewer entries than the worldwide long tail. Germany’s low overall figure results from the mass of small pages: 443,770 of the 457,225 German home pages analyzed have ranks beyond 1 million. The counts do not show where these differences come from, whether from the mix of operators, platform templates or something else.
Shops and shop platforms
Rates vary widely among German shops with a detected platform. Apart from the special case of Squarespace, Magento shops name GPTBot most often, while Wix shops almost never do. One possibility is that size and platform explain this: Magento tends to run in larger shops, Wix generates robots.txt itself, and Squarespace adds AI bots by default. I have not tested this; there is no size adjustment for each platform. The figures cover only platforms with at least 300 German home pages with rule groups.
| Shop platform | Home pages | GPTBot named | GPTBot: Disallow without Allow | ClaudeBot named | CCBot named |
|---|---|---|---|---|---|
| WooCommerce | 13,938 | 2.3 % | 1.1 % | 2.0 % | 1.6 % |
| Wix eCommerce | 7,976 | 0.1 % | 0.0 % | 0.1 % | 0.1 % |
| Shopify | 2,937 | 1.8 % | 0.3 % | 1.6 % | 1.0 % |
| Gambio | 2,201 | 1.5 % | 1.2 % | 1.2 % | 1.4 % |
| JTL Shop | 1,099 | 2.8 % | 1.8 % | 2.0 % | 1.8 % |
| Magento | 716 | 8.1 % | 5.4 % | 6.8 % | 5.9 % |
| Pimcore | 688 | 2.6 % | 2.0 % | 2.2 % | 3.3 % |
| PrestaShop | 474 | 4.9 % | 3.0 % | 5.1 % | 3.8 % |
| Squarespace Commerce | 442 | 99.8 % | 0.2 % | 99.8 % | 99.8 % |
| Shopware | 311 | 2.3 % | 1.3 % | 2.9 % | 1.9 % |
HTTP Archive, October 2026 monthly dataset (date = 2026-10-01), mobile, home pages with robots.txt rule groups, host ending in .de, shop platform according to HTTP Archive technology detection (Wappalyzer), one platform used if several were detected; only platforms with at least 300 home pages with rule groups. The generic detection “Cart Functionality” (69,806 home pages, 64,146 with rule groups, GPTBot named 8.6 %, Disallow without Allow 3.5 %) is not a platform and is omitted from the table.
Trends from February to October 2026
Comparing the raw monthly figures is misleading: the composition of the crawl changes from month to month, and some websites change their robots.txt. In the September dataset, 1.57 million home pages worldwide named GPTBot; of these, 1.25 million were still in the crawl in October, and only 790,000 still named GPTBot. Spot checks of pages that dropped the naming show disposable domains with an identical template; I have not classified them systematically. I therefore compare only the same home pages that were in the crawl in both February and October: 432,997 German home pages and 10,726,748 worldwide. Each month, the count covers the subset with rule groups (352,715 in Germany in February, 360,897 in October); shop classification and rank bucket are taken from February.
- Germany, all 3.5 % → 4.4 % · n = 360,897
- Germany, shops 5.8 % → 6.6 % · n = 78,851
- Germany, rank up to 1 million 8.9 % → 10.5 % · n = 14,097
- worldwide, all 7.6 % → 7.9 % · n = 8,650,145
- worldwide, shops 11.2 % → 11.7 % · n = 2,676,537
- worldwide, rank up to 1 million 10.8 % → 10.6 % · n = 498,050
HTTP Archive, February and October 2026 monthly datasets, mobile, home pages included in both datasets; n = home pages with rule groups in October.
In Germany, the share naming GPTBot on the same pages rises from 3.5 % to 4.4 %, and the share with Disallow without Allow from 2.4 % to 2.6 %. Among large German pages with a rank up to 1 million, it rises from 8.9 % to 10.5 %. Worldwide, naming remains almost flat (7.6 % to 7.9 %), while the share with Disallow without Allow falls from 4.3 % to 3.5 %. The counts do not show which rule changes explain this. 5,717 German pages added GPTBot between February and October, and 1,956 removed it.
The newer identifiers grow most strongly: PerplexityBot roughly doubles in Germany, from 1.0 % to 2.1 %; OAI-SearchBot rises to around 2.7 times its earlier share, from 0.6 % to 1.6 %.
Cross-check: 100 large German shops
For the readability study, I checked the robots.txt of 100 large German shops on October 11, 2026, reading 77 of them in full; the rest were blocked or returned errors. Here I can see the paths and distinguish full blocks from partial blocks:
| Identifier | named | full block | partial block | only Allow | mixed |
|---|---|---|---|---|---|
| gptbot | 11 | 4 | 2 | 3 | 2 |
| oai-searchbot | 7 | 1 | 1 | 4 | 1 |
| chatgpt-user | 6 | 2 | 1 | 2 | 1 |
| claudebot | 10 | 5 | 0 | 2 | 3 |
| perplexitybot | 9 | 3 | 1 | 3 | 2 |
| google-extended | 5 | 2 | 0 | 2 | 1 |
| ccbot | 7 | 5 | 0 | 0 | 2 |
| bytespider | 7 | 3 | 0 | 2 | 2 |
| amazonbot | 6 | 3 | 1 | 0 | 2 |
| meta-externalagent | 7 | 2 | 1 | 2 | 2 |
| applebot-extended | 5 | 1 | 0 | 2 | 2 |
Own measurement, robots.txt from October 11, 2026, 77 of 100 shops with a readable file. Full block = Disallow: / without Allow; mixed = Allow and Disallow in one entry or a block with exceptions. Shops that do not name an identifier apply their rules for * to it; those rules are not evaluated here.
11 of the 77 shops (14.3 %) name GPTBot, 4 fully block it (5.2 %), and 3 explicitly allow it. This is consistent with the HTTP Archive: large shops name AI bots more often than average, but only some block them. When CCBot and ClaudeBot are named, they are usually blocked. For OAI-SearchBot, the crawler for search in ChatGPT, four of the seven entries contain only Allow, one is a full block and one a partial block. Using its own complete rule evaluation, the readability study found OAI-SearchBot blocked in 1 of 90 shops.
Do blocked bots still come?
A robots.txt is a request, not a barrier. Only server logs can show whether a bot respects it, and even those need cautious interpretation. Of the AI identifiers, the large online shop from my study of AI assistants blocks only Bytespider (ByteDance, TikTok). In 26 days of logs, 17 requests with this identifier came from ByteDance’s network, all on one day, all for images and files, with no page fetches. That is consistent with the block being respected, but is too little to establish it; I have not separately verified whether these requests actually came from ByteDance. There were also 1,629 requests with the Bytespider identifier from other networks (AWS, Google Cloud, smaller hosting providers), whose operators I cannot identify; 713 of these belong to an actor that my log analysis classifies as an attack scanner, and 333 of its requests targeted typical attack paths. Seeing “Bytespider” in a log does not let you conclude that it is ByteDance.
GPTBot, ClaudeBot and PerplexityBot are not blocked in this shop, so I have no test for them. What I could measure is reported in the AI assistants study: for GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot and Applebot, 94.6 to around 99.9 % of requests came from the published IP ranges; for Claude-User, the figure was 66.4 %. That establishes origin, not whether a block was respected. The guide Verifying Googlebot and AI crawlers (German) explains how to check identifiers.
What this means for shops
- Most do not name GPTBot. 95.4 % of the German home pages analyzed and 93.1 % of shops do not explicitly name GPTBot; it then follows the rules for
*. If you did not choose this deliberately, you should check it. - Separate training and search. GPTBot collects for training, OAI-SearchBot for search in ChatGPT, and ChatGPT-User fetches pages live during an answer (OpenAI documentation). Among the 77 large shops, OAI-SearchBot is mostly allowed where it is named, and the readability study found a block in only 1 of 90 shops. If you want to appear in ChatGPT answers, you should do the same.
- Check templates. Turning on a setting in Cloudflare, Squarespace or your shop platform adds a ready-made group to robots.txt. Templates often block everything or nothing; you have to separate purposes yourself. My Bot Check (German) shows which rule applies to each identifier on your site.
- Check the block in your logs. A block works only for bots that respect it. Only your own logs tell you which ones do. Requests whose operators you cannot identify are controlled through bot protection rather than robots.txt.
Limitations
- Germany = .de. German operators on .com, .shop or .eu are missing, and some .de domains are based abroad. The 100 large shops also include .com domains and show a similar picture.
- Only home pages, only the mobile crawl, one measurement per monthly dataset. Robots.txt applies to the whole website, but the crawl covers only websites with enough Chrome navigations. Very small websites are missing. The figures describe this sample, not all German operators.
- Lines, not paths. In the 77 shop files, “Disallow without Allow” was a full block in 54 of 62 entries. The proportion may differ for other types of website; the indicator still counts an empty Disallow line (which allows everything) as a block, and does not capture blocks through
*. Naming is independent of this. - Shop detection. The HTTP Archive’s technology detection misses shops without a recognizable platform, includes generic cart detection and returns one platform when several are detected. The shop platform table therefore lists only platforms with at least 300 home pages.
- Templates and crawl composition. Cloudflare and platform templates, along with changing websites in the crawl, distort worldwide monthly figures in particular. The trend therefore uses the same home pages in both datasets, with a separate subset with rule groups for each month.
- One log, one blocked bot. The Bytespider finding comes from a single shop and a single month and is not sufficient evidence that the crawler respects a block.
Sources
- HTTP Archive,
httparchive.crawl.pagestable in Google BigQuery, monthly datasets February to October 2026 (date = 2026-02-01 to 2026-10-01),custom_metrics.robots_txtcolumn; queries run on October 28, 2026. Data collection FAQ. - HTTPArchive/custom-metrics, Pull Request 191 (merged February 3, 2026): robots.txt metric, counts per User-Agent.
- Google Search Off the Record: Analysing robots.txt at scale with HTTP Archive and BigQuery, April 23, 2026.
- Google: robots.txt specification, order of precedence for user agents (group principle).
- Cloudflare: Managed robots.txt (Bot Preference Sync), as of October 28, 2026.
- Squarespace: Request that AI models exclude your site, as of January 8, 2026; robots.txt of squarespace.com, checked October 28, 2026.
- Chrome UX Report: popularity buckets.
- Google BigQuery: pricing and free allowance.
- OpenAI: Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User).
- Cloudflare Radar: AI Insights, Germany, September 30–October 25, 2026 (API retrieval October 27, 2026): shares of AI bot requests by identifier (ClaudeBot 14.2 %, Googlebot 12.5 %, PerplexityBot 6.0 %, GPTBot 4.8 %, Bytespider 4.8 %, CCBot 4.6 %); shares of requests, not websites.
- Own measurements: AI readability, 100 large shops (robots.txt from October 11, 2026) and AI assistants in a large online shop (server logs September 30–October 25, 2026, anonymized).
