Whether a shop appears in ChatGPT, Perplexity or Google’s AI Overviews depends on two questions: may the bot fetch the pages, and does it find what it needs there? There is a lot of speculation about this and little measurement. On October 11, 2026, we looked at 100 large German online shops: robots.txt, bot protection, product markup, prices with and without JavaScript, llms.txt and the loading metrics from real Chrome visits. Without names, and with all the limitations of the method.
1 of 90
blocks ChatGPT search
OAI-SearchBot in the robots.txt
31 %
reject unknown bots
31 of 100 shops: error code 403 or 429, or a challenge page
38 %
provide a GTIN
anywhere in the HTML, among 56 shops with measurable product pages
61 %
respond fast enough
INP in the green range, the weakest of the three loading metrics
Own measurement, October 11, 2026. Method and limitations below.
At a glance: what was measurable across 100 shops
Each square is one shop. We could analyze the robots.txt for 90 shops, product pages only for 56. Almost a third rejected our fetch; for another 13, we found no reachable product pages via the sitemap, or only pages without Product markup.
- 56 with measurable product pages Product markup, prices with and without JavaScript analyzed
- 31 reject unknown bots 403 or 429 on robots.txt, sitemap or product page, or a challenge page
- 13 without a measurable product page no sitemap in the robots.txt, sitemap without reachable product pages, or pages without Product markup
1. In robots.txt, AI search is blocked almost nowhere
OpenAI separates its bots by purpose: GPTBot collects content for training future models, OAI-SearchBot brings pages into ChatGPT’s search results, ChatGPT-User fetches pages when someone asks about them in ChatGPT. According to OpenAI, you can block GPTBot and allow OAI-SearchBot. Anthropic separates them too (ClaudeBot for training, Claude-SearchBot and Claude-User for search and retrieval). According to Perplexity, PerplexityBot is used only for search, not for training.
- Search and retrieval (for answers in AI search)
- OAI-SearchBot 1 % (1 of 90)
- ChatGPT-User 2 % (2 of 90)
- PerplexityBot 4 % (4 of 90)
- Claude-SearchBot 1 % (1 of 90)
- Googlebot 0 % (0 of 90)
- Training (future models)
- ClaudeBot 8 % (7 of 90)
- CCBot 7 % (6 of 90)
- GPTBot 6 % (5 of 90)
- Google-Extended 2 % (2 of 90)
“Blocked” = homepage not fetchable according to robots.txt (evaluated per RFC 9309, the robots.txt standard). Purpose of the bots according to the vendors (OpenAI, Anthropic, Perplexity, Google, Common Crawl). ChatGPT-User and Perplexity-User only fetch at a person’s request; according to the vendors, robots.txt does not necessarily apply to them.
Most robots.txt files do not mention AI bots at all: GPTBot is named specifically in only 11 of 90 files, OAI-SearchBot in 7. Those that explicitly regulate AI bots usually separate training and search cleanly.
The one exception blocks all OpenAI and Perplexity bots.
So for almost all shops: if they are missing from AI answers, it is usually not because of robots.txt. Source: OpenAI, Overview of OpenAI Crawlers.
2. Almost one in three shops rejects unknown bots
For 31 of 100 shops, our fetch did not get through: the response was 403 (access denied), 429 (too many requests) or a challenge page instead of the content. We measured with an honest user agent of our own that respects robots.txt and fetched only a few pages per shop, one after another with pauses.
For 17 shops whose robots.txt we could not read live, we analyzed the most recent copy in the Internet Archive (as of March to October 2026). All blocks of the search bots and of GPTBot come from files read live; the archive copies yielded only one ClaudeBot block.
3. Product markup is there, the details are missing
For 56 shops we could measure real product pages, up to three per shop, drawn at random from the sitemap. Product markup in JSON-LD format is in the raw HTML for 49 of 56, and price and availability mostly too. It gets thin with the attributes that make a product unambiguous.
- Product markup 88 % (49 of 56)
- Price 84 % (47 of 56)
- Availability 80 % (45 of 56)
- Brand 71 % (40 of 56)
- Variants (ProductGroup) 16 % (9 of 56)
Only a page with Product markup (in the raw HTML or after JavaScript) counts as a product page. The first bar is therefore not a value for all product pages of large shops.
The GTIN (in Europe usually the EAN) is a unique, internationally standardized product identifier that helps Google classify and display products. Google lists it among the recommended properties for merchant listings. 21 of the 56 shops have a GTIN in the HTML, 11 give only the manufacturer part number (MPN), 24 neither GTIN nor MPN.
We only measured what is on the product page. Many shops also provide the GTIN, or only provide it, via a product feed, for example in Google Merchant Center; Google recommends markup on the page and a feed together. So if the GTIN is missing from the HTML, Google may still know it. Anyone who only reads the page will not find it there, though. The sample was stable: of 53 shops with two or three measured product pages, 19 had the GTIN on every page, 32 on none and only 2 on some.
Google describes variants such as size or color with ProductGroup (via hasVariant; in Google’s examples each variant has its own offer). Only 9 of 56 shops use it, although not every measured product page has variants. We did not evaluate other ways of representing variants.
Sources: Google, Merchant listing (Product) · Google, Product variants.
4. What is missing without JavaScript
Why does JavaScript matter? Because not every bot renders a page the way a browser does. In our own server logs (only fetches from the IP addresses the operators publish), it looks like this:
| Bot | Pages | JavaScript files | Loads JavaScript? |
|---|---|---|---|
| Googlebot | 226 | 238 | yes |
| Applebot | 124 | 248 | yes |
| OAI-SearchBot | 148 | 72 | partly |
| PerplexityBot | 373 | 0 | no |
| ChatGPT-User | 255 | 0 | no |
| ClaudeBot | 163 | 0 | no |
| GPTBot | 1 | 0 | no (little data) |
Own logs of evergrowsolution.de, August 3 to October 12, 2026, Googlebot only with the user agents documented by Google. Logs only show whether a bot loads the files, not whether it executes them. A small website, so an indication, not proof. On a much larger basis, Vercel at the end of 2024 concluded: the crawlers of ChatGPT and Claude partly load JavaScript files but do not execute them.
That is why we measured every product page twice: as raw HTML and as a browser displays it after JavaScript. For most shops, little changes. The gap is largest where the price is in the visible text.
- Product markup 49 → 52 of 56
- Price in markup 47 → 49 of 56
- Availability 45 → 48 of 56
- Price in text 42 → 49 of 56
- Product name in text 49 → 49 of 56
“Only in the markup”: the price is in the JSON-LD, but not in the body text. For bots that evaluate markup this is enough, for pure text analysis it is not.
For 3 shops, the price only gets onto the page with JavaScript; one of them delivers practically nothing but an empty app shell without JavaScript. That is a small share, but a real risk for these retailers: a bot that does not execute JavaScript finds no price there.
5. llms.txt: a good one in five shops has one
15 of the 16 files have the format Lighthouse expects (heading, link, at least 50 characters).
16 of the 74 shops have an llms.txt. Six deliver a normal HTML page at that address, usually an error page with status 200; anyone checking the file then sees HTML instead of text. For context: Google states that no new machine-readable files are needed for the AI features in Search. Whether other providers use the file is not established.
6. Loading metrics: responsiveness is the weak spot
For the Core Web Vitals we did not take our own measurements but queried the field data of real Chrome visits from the Chrome UX Report (mobile, whole domain, 75th percentile). Each dot is a shop, the orange line the median. Loading (LCP) and stability (CLS) are good for most (89 and 83 of 100). Only 61 of 100 pass responsiveness to clicks (INP).
LCP · Loading
89 of 100 good
INP · Responsiveness
61 of 100 good
CLS · Stability
83 of 100 good
Source: Chrome UX Report API, queried on October 11, 2026. All three metrics good: 51 of 100 shops.
For AI bots that only read HTML, loading metrics probably play a smaller role than for visitors (our assumption, not measured). We show them because they come from the same source as Google’s assessment and because INP is the metric in our data that the fewest shops pass. The values apply to the whole domain, not only to product pages.
7. By industry
We assigned the 100 shops to seven industries. The numbers per industry are small, so we show counts instead of percentages. The table is purely descriptive, not a ranking. The stronger the blue, the higher the share.
| Industry | Shops | reject bots | product pages measurable | of which GTIN | of which variants | llms.txt | CWV all good |
|---|---|---|---|---|---|---|---|
| Furniture, home, garden & DIY | 18 | 4 / 18 | 12 / 18 | 6 / 12 | 1 / 12 | 2 / 14 | 7 / 18 |
| Fashion, shoes & accessories | 18 | 4 / 18 | 10 / 18 | 3 / 10 | 0 / 10 | 2 / 14 | 6 / 18 |
| Electronics & household appliances | 17 | 5 / 17 | 8 / 17 | 4 / 8 | 3 / 8 | 3 / 12 | 7 / 17 |
| Drugstore, beauty & health | 15 | 6 / 15 | 9 / 15 | 3 / 9 | 3 / 9 | 4 / 11 | 9 / 15 |
| Sports, leisure & automotive | 11 | 5 / 11 | 6 / 11 | 2 / 6 | 1 / 6 | 1 / 8 | 8 / 11 |
| Toys, baby, books & pets | 11 | 4 / 11 | 6 / 11 | 3 / 6 | 0 / 6 | 2 / 7 | 7 / 11 |
| Marketplaces, generalists & groceries | 10 | 3 / 10 | 5 / 10 | 0 / 5 | 1 / 5 | 2 / 8 | 7 / 10 |
Two things stand out, with very small numbers: in the group marketplaces, generalists and groceries, none of the 5 measurable shops provides a GTIN. Drugstore, beauty and health has the most llms.txt files, with 4 of 11. That is not enough for rankings between industries.
What this means for retailers
- First check whether the real bots get through. robots.txt is rarely the problem. Whether bot protection hits real crawlers is only shown by the server logs: do fetches from the operators’ IP lists get status 200?
- GTIN into the markup. For 35 of 56 shops we found no GTIN in the HTML. If the EAN is already in the ERP system, in my experience this is usually a template change, not new data maintenance.
- Variants as ProductGroup. Sizes and colors with their own offer per variant, so that price and availability can be assigned unambiguously.
- Price and product name into the raw HTML. If you assemble product pages only in the browser, you should render on the server or at least deliver the core data in the HTML.
- Keep an eye on INP. For visitors and for Google, less so for AI bots. Of the three loading metrics, the measured shops miss INP most often.
How to check your own shop
- robots.txt: Is OAI-SearchBot, ChatGPT-User or PerplexityBot listed anywhere with
Disallow: /? Our AI check (in German) evaluates this for all known AI bots. - Product data without JavaScript: Open the source of a product page (not the developer tools) and search for price and product name. The product data check (in German) checks the markup of a product page.
- GTIN and variants: Is the EAN in the markup? Are sizes and colors described as ProductGroup with their own offer per variant?
- Bot protection: Check in the server logs whether the real bots get status 200 and not 403 or 429.
Method and limitations
- Selection: 100 large B2C shops in Germany by estimated organic search traffic (data source DataForSEO), excluding media, comparison portals, manufacturers without a shop and B2B retailers. Not an official ranking. Industries assigned by hand.
- Fetching: honest user agent, robots.txt respected, at most around twelve fetches per shop one after another with pauses, stop at 403, 429 or 503.
- Product pages: up to three per shop, drawn at random from the sitemap. Only a page with Product markup counts. This is a sample, not a full survey: for shops with many page templates, three pages do not cover everything.
- Selection effect: the product figures only apply to the 56 shops that allowed our fetch and had a sitemap with product pages. They say nothing about shops with strict bot protection.
- robots.txt from the archive: for 17 shops, the robots.txt comes from the Internet Archive (as of March to October 2026). It may have changed since.
- With and without JavaScript: the same pages additionally loaded in a browser without images and fonts. Pages where the browser got a challenge page do not count in the comparison.
- Core Web Vitals: field data from the Chrome UX Report (mobile, whole domain, 28 days, 75th percentile), no measurement of our own.
- Not measurable: 44 shops without a measurable product page. They are listed as “not measurable”, not as “poor”.
- Manual review: before the analysis, we re-checked samples against the saved HTML. We found and corrected several measurement errors, among them a challenge page that at first looked like a JavaScript gap, and GTINs that were only in the offers.
- Snapshot from October 11, 2026. robots.txt and bot protection change constantly.
