Getting Retrieved vs. Getting Cited by AI

Let’s picture a situation marketing teams often run into. You open the server logs and see AI crawlers coming back day after day, requesting hundreds of URLs. Then you open your AI visibility report and find the company barely appears as a source in generated answers.

At first, those two reports seem to contradict each other. They don’t. They are simply measuring different things.

The confusion usually starts because the tools marketers rely on were built to answer different questions.

• Google folds AI features into its existing reporting: appearances in AI Overviews and AI Mode are counted within overall Search Console search traffic in the Performance report rather than separated into a dedicated AI line item.
• Bing Webmaster Tools takes a different approach, with an AI Performance report showing when a site is cited in AI-generated answers across Microsoft Copilot, AI summaries in Bing, and select partner integrations.
• Ahrefs Brand Radar measures brand visibility across the AI platforms covered by its own index.
• Server and CDN logs tell you something much narrower: which URLs received requests.

And that distinction matters. Seeing an AI bot in a server log does not tell you that your page appeared as a source in someone’s answer.

That said, think about three separate events: crawling, retrieval, and citation. Once you stop treating them as different stages of the same guaranteed process, the numbers become much easier to interpret.

What are Crawled, Retrieved, and Cited Processes?

The simplest way to separate them is like this: a crawl is a request to your server, retrieval is the gathering of information while an answer is being prepared, and a citation is the visible source reference that appears in the final answer. Of those three, only crawling necessarily leaves evidence on your website.

It is also important not to lump every AI request into the same category, because even within one provider, different bots can be visiting the same website for completely different reasons. OpenAI’s own crawler documentation makes that distinction unusually clear.

• GPTBot is the training crawler. It collects content that may be used to make OpenAI’s generative models more useful and safe. A visit from GPTBot therefore tells you about potential model-training activity. It does not mean someone has just asked ChatGPT a question about your topic, or that your page is being considered for a citation.
• OAI-SearchBot is the search crawler. Its job is to discover content that can surface in ChatGPT’s search features. This is the bot that matters when thinking about whether a page can become available to ChatGPT Search. OpenAI treats this separately from model training: a site can allow OAI-SearchBot while blocking GPTBot.
• ChatGPT-User is different again because it is triggered by user activity rather than automatic crawling. When someone asks ChatGPT or a Custom GPT a question, ChatGPT may visit a specific web page using this user agent as part of fulfilling that request. In other words, this is the closest of the three to the retrieval step described above: the system is accessing a page in the context of an actual user interaction. OpenAI explicitly notes that ChatGPT-User is not used to decide whether content can appear in Search.

Those names are specific to OpenAI. Other providers document their own agents and use cases. The broader point is what matters: an AI bot request has a purpose, and different requests should not be treated as interchangeable evidence.

No documented “citation bot” arrives at your server to credit a page. Citation happens later, inside the generated answer. Your website does not receive a special request telling you that it happened.

This is why we would avoid visualizing the process as a neat funnel where every page gets crawled, then retrieved, then cited. Model training, search indexing, and answering a live user question are separate activities. They can involve different content, at different times, for different reasons.

Crawled: Page Requested

Crawling is the easiest of the three events to observe because it happens directly against your infrastructure. An agent requests a URL, and the server responds. It may return the page successfully, redirect the request, return an error, or block it altogether.

If the request succeeds, you know that the agent was able to access the content at that moment. That is useful information, but it is also where we would stop the conclusion. A successful crawl does not tell you whether the page was indexed, used in a training set, retrieved for a particular prompt, or shown as a citation later. It proves access, not visibility.

Retrieved: Content Found

Retrieval is harder to observe because it happens while the system assembles an answer. And there is more than one way for that to happen. A system might run a search and work from the results or their snippets. It might open an individual page. Or it may rely on information that was already indexed or stored during an earlier process.

That means retrieval does not necessarily produce a fresh visit to your website. A page may contribute information to an answer while today’s server log shows nothing new. A search snippet already stored in an index is an obvious example: the information can become part of the research process without the underlying page being opened again.

There is another distinction worth being strict about here. A URL appearing among search results is not the same as the page being opened, and neither observation proves that the model read the entire page. Content can enter the research process without ever becoming a visible citation.

Cited: Source Linked

Citation is the part users can actually see: a link, footnote, or source reference attached to the generated answer.

Here, too, we often see several different outcomes grouped under one metric. An AI answer may cite the company’s own website. It may cite a third-party page talking about the company, such as a directory, review site, or partner page. Or it may mention the company directly without linking to any source.

Those are very different visibility outcomes and they call for different analysis. Only the first means the company’s own content received the citation.

It is equally important not to read too much into the citation itself. A citation does not prove that the system endorses the company, recommends it, generated a click, or influenced a conversion. It proves something much more specific: the source received visible attribution in that particular answer.

Retrieval by AI Doesn’t Guarantee Citation

Let’s consider the following example of a ChatGPT conversation that shows how wide that gap can become. The prompt was deliberately ordinary: “best cat food in France.” It had no connection to the website being examined, which makes it easier to look at the mechanics without trying to explain the results through a particular brand’s performance.

The conversation produced three search rounds. The first listed 12 pages and marked 2 as cited. The second listed 37 pages and cited 2. The third listed 17 pages and cited 3. Across the conversation, that amounted to 7 source references from 64 unique URLs.

One domain makes the pattern especially interesting, although there is an important caveat. In the second round, ziggyfamily.com accounted for 12 of the 37 listed pages and received no citations. In the third round, all 17 listed pages came from the same domain, and three were cited.

A search round returning pages from only one domain is consistent with a site-scoped search. Because the query text itself is unavailable in the saved session, we would not compare those two rounds as though they were conducted under identical conditions. The safer comparison is the aggregate from the two rounds that appear unrestricted: 49 pages listed and 4 citations.

The comparison comes from the pages listed in the recorded search results and the source references visible in the same saved conversation. To inspect them, we used Orit Mutznik’s ChatGPT Fanout Explorer, a free, read-only bookmarklet that reconstructs the fan-out behind a prompt using data available inside the conversation in the browser.

There are two limitations we would keep firmly attached to these numbers. First, “listed” does not mean “read.” The fetched label refers to rows recorded in the search results; it does not prove that every page was opened in full or that every URL received a verified server request. Second, this is one observed conversation, not a citation benchmark for a website, an industry, or ChatGPT as a whole.

What Logs Actually Show

Server and CDN logs are still extremely useful here, as long as they are used for the question they can actually answer. Depending on the configuration, they may show requested URLs, timestamps, HTTP response statuses, user-agent strings, and IP addresses.

We would start by verifying bot identity rather than trusting the user-agent string alone, because user agents can be spoofed. Where the provider publishes the necessary information, check documented IP ranges or reverse DNS. Then separate successful requests from blocked and failed ones.

From a successful request, one conclusion is safe: the resource was accessed at that time.

Logs can show which bot requested a page, which URLs it visited, and whether those requests succeeded. That makes them useful for checking technical readiness for AI crawling, including server responses, caching, CDN behavior, and blocked or failed requests.

They can also reveal patterns in how GPTBot, OAI-SearchBot, and ChatGPT-User interact with different parts of the site. If certain pages are visited more often, that can support hypotheses about which content attracts more bot activity.

What logs cannot prove is citation. An increase in ChatGPT-User visits does not mean citations increased by the same amount, and one bot visit does not equal one citation. Logs work best for understanding technical accessibility and bot-content interaction, not for measuring AI visibility directly.

Uncited by AI Doesn’t Mean Unseen

The opposite mistake happens just as often. A team opens an AI visibility tool, sees almost no citations, and concludes that AI systems are not finding the website at all.

The saved ChatGPT session above shows why that conclusion is too strong. A substantial number of pages appeared during retrieval, while only a small subset ended up cited.

Every citation platform also has its own field of view. Its results depend on which platforms it monitors, which prompts it checks, how frequently it checks them, and the time period covered. Bing’s AI Performance report, for example, covers Microsoft Copilot, AI-generated summaries in Bing, and select partner integrations. Ahrefs notes that Brand Radar’s visibility index may have limited coverage for brands with little or no search demand.

A drop in citations does not necessarily mean AI bots can no longer see or access the site. It may reflect the limits of the tracking system itself, since Ahrefs, Bing, and Google each measure AI visibility differently and with their own coverage gaps.

It is also possible for bots to crawl or retrieve a page and still leave it uncited in the final answer. When access is working, but citations remain low, the next question is not always technical. It may be about the content itself: how well it matches the query intent, how specific and useful it is, and whether it gives the system a strong enough reason to reference it.

So low citation numbers should not automatically trigger a technical audit. Sometimes the site is accessible, and the real issue is whether the content deserves the citation.

AI Bot Visits Aren’t the Finish Line for Marketers

Frequent bot visits are a good sign, but they are only the beginning. If requests are being blocked, throttled, redirected incorrectly, or returning errors, the technical side needs attention, and the crawl and indexing settings that govern that route should be checked first.

Once bots can access the site reliably, the focus shifts to content strategy: understanding user questions, analyzing which competitors earn citations, and identifying what makes their content more relevant, useful, or authoritative.

The questions to investigate are:

• Which questions do prospective customers actually put to these systems?
• Which sources currently receive citations for those questions?
• What evidence or specificity do those cited sources offer?
• Where could existing content be more useful, more specific, more current, or better supported?

That is where content strategy, competitor analysis, brand positioning, and content optimization come in. That is where content strategy, competitor analysis, brand positioning, and content optimization come in. Technical accessibility creates the possibility of being cited, but it does not create the citation itself. Progress should be measured through AI search visibility over time rather than bot traffic alone.

If AI bots can access the site reliably, the website is technically ready. The next task is making the content worth citing.