Claude can mention your brand in two ways: from what the model learned in training, and from live web search, where it shows citations with links. To be cited in search answers, let Anthropic’s search and user-fetch crawlers (Claude-SearchBot and Claude-User) reach your site, publish pages that answer your buyers’ questions directly with checkable facts and named sources, and earn mentions on other sites Claude is likely to find. Then measure it with a fixed set of prompts and your server logs.

How Claude finds and cites web pages

Web search with citations. Claude can search the web when a question needs current information, and Anthropic’s help center says every response that uses search includes citations so users can check the sources. Web search counts toward usage limits on free accounts, so it is available beyond paid plans. On Team and Enterprise plans an admin must enable it for the workspace. In the newer Claude experience there is no search toggle; Claude decides when to search.

Web fetch. When a user pastes a URL, Claude can retrieve that page and read it in full.

Training knowledge. Without search, Claude answers from what its models learned during training. Brands that are widely and consistently described across the web are more likely to be known. You can’t update this directly; it changes when new models are trained.

Anthropic’s three crawlers, and what to allow

Anthropic documents three bots, each controlled separately in robots.txt:

BotWhat it doesIf you block it
ClaudeBotCollects public web content that may be used for model trainingYour future content is excluded from training data
Claude-SearchBotIndexes content to improve Claude’s search resultsYour pages may be less visible in Claude’s search answers
Claude-UserFetches pages when a user’s question or link requires itClaude can’t retrieve your pages for users

If you want to be cited in Claude’s search answers, Claude-SearchBot and Claude-User need access. You can still opt out of training. For example:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Older guides mention “Claude-Web” or “anthropic-ai”. Those names do not appear in Anthropic’s current documentation, so blocking or allowing them does nothing useful. Anthropic says it honors robots.txt, supports Crawl-delay, and warns that blocking by IP address may not give a reliable opt-out. Also check that your CDN or firewall isn’t blocking these bots: Cloudflare, for instance, lists ClaudeBot, Claude-SearchBot and Claude-User as separate verified bots that can be allowed or blocked.

What makes a page worth citing

There is no published Claude ranking algorithm, so treat anyone who claims to know Claude’s exact citation weights with suspicion. What is well supported:

1. Answer the question in the first lines

Put a direct, complete answer to the page’s main question in the first one or two sentences, then the detail. Search-grounded answers quote or summarize short passages, and a self-contained answer near the top is easier to lift accurately.

2. Add statistics, quotations and sources

The GEO research paper by Aggarwal et al. (KDD 2024) tested content changes on generative search responses and found that adding relevant statistics, quotations from credible sources, and citations to sources were among the most effective ways to increase a source’s visibility, by up to about 40% in their tests. Use real numbers with a named source and date, and link to the original.

3. Make your entity unambiguous

Use one consistent company name, describe what you do in plain words on your homepage and About page, and keep the same facts (name, location, category, founders, products) consistent across your site, LinkedIn, directories and review sites. Organization schema with sameAs links to your official profiles helps any system connect those references.

4. Earn mentions on other sites

When Claude searches, it reads whatever pages the search returns for the question, which are often comparison articles, reviews, forums and industry publications rather than your own site. Being accurately described on those pages matters as much as your own content. It also feeds the training knowledge of future models.

5. Keep pages accurate and fast to fetch

Claude-User fetches pages live, so pages that load slowly, block bots, hide content behind scripts or carry out-of-date facts are weaker sources. Update facts when they change, and only change a page’s date when you have actually changed the content.

How to check whether Claude cites you

  1. Run a fixed prompt set. Write 20–50 questions your buyers ask. Run them in Claude with web search on, and record whether you’re mentioned, whether you’re linked, and which sources are cited. Repeat monthly with the same prompts, and run each more than once, because answers vary.
  2. Check your server logs. Look for the Claude-SearchBot and Claude-User user-agent tokens. Search-bot visits show you’re being indexed; user-bot visits show Claude fetching your pages in response to real questions. Anthropic publishes its crawler IP ranges so you can verify requests are genuine.
  3. Track referrals. Visits from claude.ai in your analytics are people clicking through from Claude’s citations.
  4. Use a tracking tool if you need scale. Several AI visibility tools include Claude, some only as a paid add-on; see our AI visibility tools comparison.

Claude compared with other engines

The fundamentals (crawl access, clear answers, credible sources, third-party mentions) are the same across engines, but the crawlers differ. ChatGPT search depends on OpenAI’s OAI-SearchBot; Perplexity uses PerplexityBot and Perplexity-User; Google’s AI Overviews and AI Mode draw on Google’s search index. A robots.txt rule that blocks one company’s search bot doesn’t affect the others, so check each. For the other engines see How to Get Cited by ChatGPT, How to Rank in Perplexity AI and How to Optimize for Google AI Overviews.

What to do this week

  1. Check robots.txt and your CDN/firewall settings for ClaudeBot, Claude-SearchBot and Claude-User, and decide deliberately what to allow.
  2. Search your server logs for the three Claude user-agent tokens to see whether they reach you.
  3. Write 20 buyer questions and run them in Claude with search on. Note who gets cited.
  4. Rewrite the opening of your three most important pages so the first two sentences answer the page’s main question.
  5. Replace unsourced claims on those pages with sourced statistics or remove them.

FAQ

Does Claude search the web?

Yes. Claude can search the web when a question needs current information, and responses that use search include citations. Anthropic’s help center describes web search for free accounts as well as paid plans; on Team and Enterprise plans an admin must enable it. Claude can also fetch a specific page when a user shares a link.

Which Anthropic crawlers should I allow?

To be cited in Claude’s search answers, allow Claude-SearchBot (search indexing) and Claude-User (user-requested fetches). ClaudeBot is the training crawler; you can block it without blocking the other two.

Does blocking ClaudeBot remove me from Claude’s answers?

No. ClaudeBot only affects whether your future content may be used in training. Search indexing and live fetches use Claude-SearchBot and Claude-User, which are controlled separately.

Does content that works for ChatGPT or Perplexity also work for Claude?

Mostly. Clear answers, sourced facts and third-party mentions help across engines. The main Claude-specific step is making sure Anthropic’s search and user crawlers can reach your pages.

How do I measure whether Claude is citing my brand?

Run a fixed set of buyer questions in Claude with search on each month, check your logs for Claude-SearchBot and Claude-User visits, and track referrals from claude.ai in your analytics.

Sources

Related reading: What is Generative Engine Optimization?, How to Measure AI Citations, Best AEO Tools in 2026.