Server log analysis: AI assistants in a large online shop – fetches, visits and purchases

In the study of the ChatGPT utm outage, I showed how quickly visits from ChatGPT can disappear from web analytics. Here I go a step further. I am interested in three questions: Which AI bots fetch the shop, and what do they read? Does ChatGPT tend to answer from its own search index or with fresh page fetches? And do people arriving via AI assistants buy differently from visitors from Google Search? The overview of my studies on SEO and AI search places this measurement alongside the others.

Data and method

The basis is the server logs of a large online shop from September 30 to October 25, 2026, analyzed by calendar day in the shop’s local time. September 30 contains only the evening and is therefore excluded from all comparisons over time. I also use web analytics and Google clicks per product from the same shop’s Search Console. I separate three levels:

• Fetches: requests by AI bots, recognized by their identifier (User-Agent). Where the provider publishes IP ranges, every request is checked against them. A fetch is not a human visit. I call ChatGPT-User and Claude-User requests live fetches: they retrieve a page while someone asks a question in a chat.
• Visits: page requests without a bot identifier whose URL carries an AI tag, for example utm_source=chatgpt.com. A visit starts with the first page request per IP address and AI source after a gap of more than 30 minutes. This is an approximation, not a counted person or a verified click. Using a gap of 5 or 60 minutes changes the total number of visits by less than 8 %.
• Purchases: from web analytics, only consented visitors, with the purchase in the same session. The analysis covers October 1–7 and 16–25, 2026. I exclude the week of the utm outage in between. Web analytics is not linked to individual log entries.

I assign page types by URL pattern: product pages, categories, guides and magazine, homepage, images and files, and other. The log extract contains the recognized AI bots and requests with AI tags; I do not have the shop’s complete normal navigation traffic. I do not name the shop. I report only shares and ratios instead of absolute numbers.

Who fetches the shop

Around 1.1 % of all requests to the shop in these 26 days came from AI bots. I only have a rounded total for all requests, so this value indicates the scale. More than half of AI fetches come from meta-externalagent, followed by OAI-SearchBot, the bot for search in ChatGPT, and PerplexityBot.

Live fetches, counted by the ChatGPT-User and Claude-User identifiers, together account for only 3.7 % of AI fetches. The vast majority are crawlers that collect pages in advance: for search indexes, such as OAI-SearchBot and PerplexityBot, or for model training, such as GPTBot and ClaudeBot, as described by OpenAI and Anthropic.

Genuine, or just using the same name?

Anyone can put a bot identifier in a request. OpenAI, Anthropic, Perplexity, Apple and Amazon therefore publish lists of their IP ranges. For most bots, almost every request comes from those ranges:

Two values are lower. For OAI-SearchBot, 94.6 % of requests came from the published ranges. The remaining requests fetched almost exclusively images and files, hardly any pages. For Claude-User, it was 66.4 %. Anthropic publishes a single list for all its bots with 14 entries, most of them individual addresses (as of October 20, 2026). Some genuine live fetches may come from networks not listed there. I therefore refer to requests outside the published IP ranges, rather than fake requests. For Amazonbot, 99.7 % of requests came from Amazon’s list (as of September 8, 2026); the small remainder comes mainly from Google Cloud networks. Meta’s crawler page publishes no IP list for meta-externalagent (as of October 27, 2026). I therefore checked the network: more than 99.9 % of requests came from Meta’s network (AS32934). This is weaker than checking against a published list. However, fetches with a Meta identifier from other networks are virtually absent in this shop.

Before and after October 8

According to Search Engine Journal, GPT-6 also became available on free ChatGPT plans on October 8, 2026. I compared fetches per hour before and from that day onward, using only requests from the published IP ranges. This describes a change without establishing a cause: other things also change over three weeks.

Server logs of a large online shop, September 30–October 25, 2026. Before: October 1–7; from October 8 onward: October 8–25. Second column: seven days each with the same mix of weekdays before and after the cutoff.

After the cutoff, OAI-SearchBot fetched around 2.2× as often per hour as before, and PerplexityBot 1.6×. ChatGPT-User was at 0.74, GPTBot at 0.63. With the same mix of weekdays, the picture remains similar for OAI-SearchBot, PerplexityBot and ChatGPT-User. For GPTBot, it reverses: the ratio is 1.1. The decline for GPTBot is therefore not a robust finding.

What AI reads

The bots read very different parts of the shop. Live fetches mainly go to product pages: 83.0 % for ChatGPT-User and 77.1 % for Claude-User. Guides and magazine play little part, at no more than 4.6 %. Crawlers, by contrast, fetch many images and files: 70.1 % for ClaudeBot, 55.5 % for meta-externalagent and 45.5 % for OAI-SearchBot.

Server logs of a large online shop, September 30–October 25, 2026. For OpenAI, Anthropic, Perplexity and Apple bots, only requests from the published IP ranges; Amazonbot and meta-externalagent counted by identifier. Rounding may cause row totals to differ from 100. “Other”: homepage, search, APIs and URLs without a fixed type, including error pages.

A fetch shows that the bot requested a URL. Logs do not show whether it processed the content or used it in an answer.

Where bots encounter redirects

Some bots request URLs in a different form from the one used by the shop. The shop then responds with a redirect; if the bot follows it, it needs at least one more request. This happens most often with Claude-User:

For Claude-User, more than half of live fetches receive a redirect. In 93 % of these cases, the URL is missing its trailing slash. The bot is therefore requesting the URL without a trailing slash. For Applebot, almost all redirected URLs contain uppercase letters; for Amazonbot, just under four in five, often together with parameters. Uppercase letters can also be valid in product URLs, however, so the cause remains open there. ChatGPT-User and OAI-SearchBot rarely encounter redirects (2.2 % and 1.4 %).

Index or live fetch

ChatGPT can know a product page in two ways. OAI-SearchBot collects pages in advance for the search index. ChatGPT-User fetches a page live while someone asks a question. I linked every ChatGPT visit to a product page with earlier fetches of the same page.

The analysis covers visits from October 16 to 25, after the utm outage of October 8–15, which I exclude. All successful fetches from the published IP ranges since September 30 that occurred strictly before the visit count as prior fetches.

For 92.8 % of ChatGPT visits to product pages, OAI-SearchBot had previously fetched the same page. That sounds like a clear finding, but it is not. In 26 days, OAI-SearchBot fetched 69.8 % of all active products and 99.0 % of the products with the most Google clicks. Almost every product people asked about had therefore already been fetched anyway. At 92.8 %, ChatGPT visits are even slightly below the value for all products clicked on Google (96.6 %). This does not support selection through the index.

The second value is more reliable: only 6.5 % of visits had a live fetch of the same page by ChatGPT-User in the preceding hour. Counting all tagged product page requests rather than only the starts of visits gives 7.2 %. Among visits with a prior OAI-SearchBot fetch, that fetch was rarely within the preceding hour (1.7 %), and usually more than a day earlier (56.5 %).

And when was the bot last there? For both bots, I measured how long before the visit the last successful fetch of the same product page occurred:

Server logs of a large online shop, September 30–October 25, 2026. October 16–25, 2026, excluding the days of the utm outage; only successful fetches from the published IP ranges that occurred before the visit. Fetches before September 30 are unknown.

For a third of visits, ChatGPT-User had not fetched the page at all since September 30; for just under half, its last fetch was more than a day earlier. Live fetches do occur, but usually days before the visit and therefore for other conversations. This suggests that ChatGPT reuses earlier fetches and index data; logs cannot prove it.

One indication of such other routes: 36.1 % of ChatGPT visits carried feed or shopping markers in the URL. This suggests shopping links; the logs do not show whether they were product cards or whether the shop appeared there through a feed.

More purchases per session from ChatGPT

For purchases, I use web analytics. I compared sessions arriving via ChatGPT or Gemini with sessions from organic Google Search. Purchases in the same session are counted, only for consented visitors, on October 1–7 and 16–25, 2026. Instead of purchase rates, I report only the ratio to organic Google.

ChatGPT sessions had around 1.5× as many purchases per session as organic Google sessions. When the session starts on a product page, the ratio is around 2.3×. Gemini shows the same direction (1.4 and 2.0), but with a small sample and therefore more uncertainty. Revenue per session from ChatGPT is only 1.17× that of organic Google. An individual purchase from ChatGPT is therefore smaller on average.

75.5 % of ChatGPT sessions start on a product page, compared with 57.8 % for organic Google. ChatGPT sessions therefore start on product pages more often. Even within this landing page type, purchases per session are higher. I did not measure whether different search or purchase intentions contribute to this.

Perplexity, Claude and Copilot bring too few sessions during these days to report a ratio. Perplexity is also above organic Google; Claude and Copilot have only isolated purchases. These figures do not establish a cause. They describe who arrived by which route and what happened in the same session.

AI and Google: the same products?

I compared products with at least one ChatGPT visit with products that had at least one Google click in Search Console. The overlap is moderate: the Jaccard index is 0.34. Of all products appearing in at least one of the two lists, around a third therefore appear in both.

• 84 % of products visited via ChatGPT also had Google clicks. ChatGPT therefore mainly sends visitors to products that are also in demand in Google Search.
• 16 % had no Google click at all in Search Console’s 90-day period (July 16–October 13). “No click” means no click in the Search Console data, not unknown to Google.
• Conversely, only just over a third (36 %) of products clicked on Google also received a ChatGPT visit. Google’s longer measurement window and greater recorded traffic volume contribute to this.

The periods do not coincide: Google clicks from July 16 to October 13, ChatGPT visits from September 30 to October 25. Counting only ChatGPT visits up to October 13, 13.7 % of products have no Google click. The scale therefore remains similar.

Compared with a small website

Is this typical of large shops or of all websites? As a cross-check, I analyzed the same metrics for evergrowsolution.de over the same period: a small website with guides and tools, without products. Its logs contain only truncated IP addresses. Authenticity can therefore be checked only at network level there, and shares based on few requests are uncertain.

Two things stand out. First, AI bots’ share of requests is higher on the small website. These metrics do not show why; a plausible but untested explanation is that it has few human visitors while the bots still come. Second, Claude-User hardly encounters redirects there, although there are only 13 requests. In the shop, many redirects involve URLs without a trailing slash. This points more to a feature of the shop’s URLs than to the bot’s behavior, but with so few requests the two cannot be reliably separated. Both sites have this in common: meta-externalagent is the largest AI crawler, and ChatGPT-User fetches almost only pages, not images or files.

Worldwide, Germany, large shop, small website

Is the shop an unusual case? For context, I retrieved public data from Cloudflare Radar for the same period: how many requests AI bots make to websites behind Cloudflare, worldwide and with the Germany location filter. The same seven bots are compared (ClaudeBot, GPTBot, OAI-SearchBot, PerplexityBot, ChatGPT-User, Amazonbot, Applebot). I leave meta-externalagent out of the time series: it accounts for more than half of AI requests in the shop and would dominate the series on its own.

Worldwide, the same bots mostly stay between 85 and 120 % of the baseline. In the shop, they are 40 to 100 % above it on many days from October 7 onward, driven by OAI-SearchBot, and more than three times as high on October 17. Values in Germany fluctuate more and are also roughly twice the baseline from October 17 to 21. All three series rise together on October 17, but the shop rises most. The shop therefore shows additional spikes absent from the worldwide series. The first sharp rise is on October 7, the day GPT-6 launched on paid plans; free plans followed from October 8. The logs do not establish a cause.

Who crawls: the same eight bots compared

Share of requests by these eight bots, September 30–October 25, 2026. Worldwide and Germany (location filter): Cloudflare Radar. Shop and small website: server logs, all requests with the respective identifier.

In the shop, meta-externalagent is even more dominant than average, and the search bots OAI-SearchBot and PerplexityBot take a larger share than worldwide. ClaudeBot and GPTBot, which their operators say collect mainly for training, are much less common in the shop. According to Cloudflare, training accounts for the largest individual share of AI requests worldwide (44 %), with another 39 % from bots with mixed purposes; only 4 % are live fetches for users, compared with 13 % in Germany. Commerce is the most frequent target: it accounts for 29 % of AI bot requests worldwide, more than any other individual industry.

How many fetches per visit?

Cloudflare measures differently from me: it counts page requests identifiable through their referrer as clicks from the platform; I count visits with utm tags. Clicks from apps may arrive without a referrer and then be missing from both counts. These figures cannot establish an exact factor, but the gap is large: in traffic covered by Cloudflare, there were around 861 OpenAI bot page fetches per recognized page request via a ChatGPT link; in this shop, around 25 per tagged visit. Up to October 7, the figure was 14; from October 16 onward, 37: estimated daily fetches were higher afterwards, and tagged daily visits lower. The logs cannot separate fewer visits from more visits without a tag (see the utm outage). For comparison: worldwide, Anthropic has 486 fetches per recognized page request, and Perplexity 3,934.

How often do shops disallow AI bots in robots.txt?

Worldwide: Cloudflare Radar, robots.txt of the most visited domains in the Ecommerce category, fully or partly disallowed, as of October 2026. Germany: study of AI readability across 100 online shops, 90 evaluable robots.txt files (73 fetched live, 17 from the Internet Archive, dated March–October 2026), counted when the homepage is disallowed for the bot.

In both selections, robots.txt disallow rules for these five bots are the exception, and training bots are excluded more often than bots for search and live fetches. The columns are not directly comparable: Cloudflare also counts individual disallowed paths, while my study counts only disallowed homepages. A robots.txt rule is also only a request; bot protection can reject bots even without an entry. For shops, this matters: according to OpenAI, pages that block OAI-SearchBot do not appear in ChatGPT search answers, at most as navigation links. In this shop, OAI-SearchBot had previously fetched the page for almost all ChatGPT visits to product pages.

What others measure

For context, here are the main public figures from other data sources. They use different populations and methods, so they provide context rather than a direct comparison.

• meta-externalagent is also large elsewhere: At Fastly, Meta accounted for 52 % of AI crawler requests from mid-April to mid-July 2025 (Fastly, August 19, 2025). Cloudflare saw GPTBot in first place and Meta in third in July 2025 (Cloudflare, August 29, 2025). The ranking therefore depends on the network and period.
• Heavy AI traffic in commerce: Akamai sees particularly high AI bot traffic in commerce, with over 25 billion requests in July and August 2025; retail accounts for three quarters of detections (Akamai, Fraud and Abuse Report 2025).
• Do AI visitors buy more often? For US retailers in July 2025, Adobe still measured a purchase rate 23 % lower for visits from AI services (Adobe, August 21, 2025), then 31 % higher than other sources during the 2025 holiday season (Adobe, January 12, 2026), and 60 % higher in retail in July 2026 (Adobe, August 19, 2026). An analysis of 973 shops using 2025 data, by contrast, finds around 13 % higher odds of purchase for organic search visitors than for ChatGPT visitors in its main adjusted model; the difference was not significant in several robustness checks (Kaiser and Schulze, Marketing Science, 2026). Comparisons therefore depend heavily on the period and data basis. This shop’s result is in line with the more recent Adobe figures, but likewise does not establish a cause.
• The identifier alone is not enough: On protected media sites, the median share of fetches claiming to be GPTBot that failed verification was high. Across all GPTBot fetches in the panel combined, it was only 2 %, because genuine fetches were concentrated on a few large websites (Centinel Analytica, 2026). How many such fetches appear therefore depends heavily on the website. Centinel uses broader checks than I do; I checked identifiers against the published IP ranges where available and do not treat a missing match as proof of spoofing.
• The shift towards search bots is not new: A monthly report based on Cloudflare Radar data shows that from January to September 2026 OAI-SearchBot's share of worldwide AI bot traffic rose from 1.7 to 3.6 % and PerplexityBot's from 0.2 to 2.2 %, while GPTBot fell from 12.6 to 6.2 %. Of OpenAI's requests, 45 % came from its search and live-fetch bots in September, up from 19 % in January. Retail was the largest target industry at 25.7 % (SEOmator, 10 October 2026). The shift towards OAI-SearchBot that this shop shows in October fits that picture.

What shops can take from this

The figures come from one shop. I derive my recommendations from them, but they apply generally rather than only to this shop. In order of importance:

1. Keep product pages accessible to AI providers’ search indexes. For almost all ChatGPT visits to product pages, OAI-SearchBot had fetched the page in advance; live fetches were rare. According to OpenAI, pages that block OAI-SearchBot do not appear in ChatGPT search answers. This also applies if bot protection rejects it unintentionally. The guide GPTBot, OAI-SearchBot, ChatGPT-User (German) explains which OpenAI bot does what.
2. Do not block live fetches. Product pages account for 77 to 83 % of ChatGPT-User and Claude-User requests, made while someone asks about them in a chat. A block or strict rate limit affects those moments directly. OpenAI notes that robots.txt rules may not apply to ChatGPT-User; in that case, only server or bot protection rules help. According to Anthropic, Claude-User can be controlled through robots.txt.
3. Handle redirects for alternative URL forms cleanly. Bots request URLs without a trailing slash or with different spelling. A single redirect straight to the valid URL, without chains, costs them only one additional request.
4. Measure the AI channel with a second signal. Alongside web analytics, count live fetches and tagged visits in the server logs. Web analytics sees only consented visitors, and the tag can fail, as it did in October 2026.
5. Verify bots using the official IP ranges rather than the User-Agent. The identifier alone proves nothing. If a list is very small, however, “outside the list” is not proof of spoofing. The guide Verifying Googlebot and AI crawlers (German) explains how to check.

Limitations

• Comparison data with a different method: Cloudflare Radar counts requests to websites behind Cloudflare and identifies bots in its own way. Trends and shares therefore provide context; differences in populations, detection, day boundaries and denominators limit their comparability. Cloudflare does not describe in detail exactly what the Germany location filter covers. The external studies have different populations.
• One shop, 26 days: The values describe this shop during this period. Other product ranges, sizes and countries may look different. The small website is a cross-check, not a representative sample.
• Visits are approximations: They rely on an IP address, a tag and a 30-minute gap. Multiple people behind one address are combined; changing addresses splits a visit. Untagged visits are entirely absent.
• Logs do not show answers: Matching index and live fetches establishes a temporal correspondence for the same URL. Fetches before September 30 and other data sources are unknown.
• Purchases only with consent: Web analytics is not extrapolated. The ratios describe groups, not a cause. Gemini has few purchases; Perplexity, Claude and Copilot have too few for a ratio.
• Unequal periods: Google clicks and ChatGPT visits come from different periods.
• Authenticity based on available lists: Checked against the published IP ranges at the time of analysis (Amazon: as of September 8, 2026; newer addresses may be missing). No list is published for meta-externalagent; only Meta’s network is checked. Both are counted by identifier in shares and time series.
• Comparison over time without causality: Different days, seasonality and provider changes cannot be separated.
• Page types and redirects: Classification follows URL patterns. Redirect destinations are not in the logs.

Sources

• Own analysis: server logs of a large online shop, September 30–October 25, 2026; web analytics of the same shop, October 1–7 and 16–25, 2026; Google clicks per product from Search Console, July 16–October 13, 2026. Anonymized, shares and ratios only.
• OpenAI: Overview of OpenAI Crawlers (OAI-SearchBot, GPTBot, ChatGPT-User, IP lists).
• Amazon: Amazonbot (IP list, as of September 8, 2026). Meta: Meta Web Crawlers (identifiers, no IP list).
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (ClaudeBot, Claude-User, Claude-SearchBot, IP list).
• Perplexity: Perplexity Crawlers (PerplexityBot, Perplexity-User, IP lists).
• Apple: About Applebot (verification through DNS and an IP list).
• Matt G. Southern: ChatGPT Traffic Drop Appears To Be A Tracking Problem, Search Engine Journal, October 15, 2026 (launch of GPT-6).
• Cloudflare Radar, AI insights and robots.txt data: radar.cloudflare.com, retrieved through the Radar API on October 27, 2026.
• Cloudflare: The crawl-to-click gap, August 29, 2025. Fastly: Threat Research Q2 2025, August 19, 2025. Akamai: Fraud and Abuse Report 2025.
• Adobe Digital Insights: Generative AI-powered shopping rises, August 21, 2025; AI-driven traffic surges across industries, January 12, 2026; U.S. consumers are embracing LLMs, August 19, 2026.
• Kaiser, M. and Schulze, C.: ChatGPT Referrals to E-Commerce Websites, Marketing Science, 2026.
• SEOmator (Emre Elbeyoglu): Monthly AI Crawler Report: September 2026, 10 October 2026, data from Cloudflare Radar, January to September 2026.
Centinel Analytica: Fake Googlebot and AI crawlers: 2026 data, September 22, 2026, updated September 30, 2026.
• Sources retrieved on October 26 or 27, 2026; data dates as specified for each source.