AEO and GEO for car dealers

Should your dealership block AI crawlers? Search bots, training bots and the robots.txt lines

By Michael Garber, Co-founder, AutoLander · Updated October 31, 2026

Short answer

Should I block AI crawlers? For a car dealership, keep the AI search and retrieval bots allowed, because they are how ChatGPT, Claude and Perplexity read and cite your store. Blocking training bots such as GPTBot and ClaudeBot is a separate business choice, since OpenAI and Anthropic run training apart from search. Google-Extended needs more thought: Google says it does not affect inclusion in Google Search, but it also covers grounding in Gemini Apps. Write the rules bot by bot, and never block every crawler at once.

Should a car dealership block AI crawlers?

No, not the ones that matter for being found. A dealership should keep AI search and retrieval bots such as OAI-SearchBot, Claude-SearchBot, Claude-User and PerplexityBot allowed so assistants can read and cite its pages. Blocking training bots is a separate business choice, because OpenAI and Anthropic run training on different crawlers from search.

The reasoning is simple for a store. Your website exists to put your name, your inventory and your service department in front of local buyers, and an assistant can only name or cite what its crawler was allowed to read. Anthropic says blocking Claude-SearchBot prevents indexing for search and reduces visibility in its search results, and that blocking Claude-User prevents retrieval when a user asks a question. Blocking those bots amounts to asking not to be read.

Before deciding anything, check whether ChatGPT and Claude can read your website today, because a robots.txt file is easy to forget once it is written. Crawler access is the base layer of AEO for car dealerships: the answer pages, reviews and listings work that follows only counts if the assistants can get in.

Which AI bots are search bots and which are training bots?

The major AI companies publish the names of their crawlers and what each one does. Search and retrieval bots read pages so an assistant can find, quote and cite them, while training bots collect content that may train future models. The table lists the ones a dealer will see, as each company describes them.

Company	Crawler or token	Job	What to know before blocking
OpenAI	OAI-SearchBot	Search: surfaces sites in ChatGPT search, not used for training	OpenAI says sites must allow it to be eligible for ChatGPT search results
OpenAI	GPTBot	Training: content that may train OpenAI’s models	Disallowing it signals your content should not be used in training
OpenAI	ChatGPT-User	User actions: certain actions a ChatGPT user starts	OpenAI says robots.txt rules may not apply to it
Anthropic	Claude-SearchBot	Search: improves Claude’s search results	Anthropic says blocking it prevents indexing for search
Anthropic	Claude-User	Retrieval: fetches pages when a Claude user asks	Anthropic says blocking it prevents retrieval for user questions
Anthropic	ClaudeBot	Training: content that may train Anthropic’s models	Blocking it signals exclusion from future training data
Perplexity	PerplexityBot	Search: surfaces and links sites in Perplexity, not used to train foundation models	Respects robots.txt; changes can take up to 24 hours
Perplexity	Perplexity-User	User actions: visits a page to answer a user’s question	Perplexity says it generally ignores robots.txt
Google	Google-Extended	A robots.txt token for Gemini training and grounding in Gemini Apps and Vertex AI	Google says it does not affect inclusion in Google Search and has no separate user agent
Apple	Applebot	Crawler whose data feeds Apple search features in Spotlight, Siri and Safari	Separate from Apple’s training opt-out
Apple	Applebot-Extended	Opt-out token for training Apple’s generative models	Apple says it does not crawl pages itself

Descriptions paraphrase each company’s own crawler documentation as of September 2026. Sources are listed at the end of this article.

What happens if your site blocks OAI-SearchBot?

Your site drops out of eligibility for ChatGPT search results. OpenAI says a site must allow OAI-SearchBot to crawl it, and must let OpenAI’s published search bot IP addresses through its host or CDN, to be eligible for inclusion. Of every line in a dealer’s robots.txt, this is the one to get right for ChatGPT.

There is a wrinkle. OpenAI’s publisher FAQ says that if a page is disallowed but OpenAI learns its URL from a third-party search provider or other pages, ChatGPT Atlas may still show the bare link and title. A noindex meta tag prevents that, and the crawler has to be allowed in to read the tag. So robots.txt controls reading, while noindex controls listing. For a dealer that wants to be found, the takeaway stays the same: allow OAI-SearchBot and give it pages worth quoting.

Being readable is the entry ticket, and what ChatGPT does with your pages depends on much more. For the rest of the picture, read how ChatGPT chooses a dealership when local buyers ask for one.

Does blocking Google-Extended remove you from AI Overviews?

No, based on how Google describes its controls. Google says Google-Extended governs whether its crawled content trains future Gemini models and grounds answers in Gemini Apps and Vertex AI, and that it does not affect inclusion in Google Search. Google AI Overviews and Google AI Mode sit under Search’s own controls.

Google’s AI features page says the controls for how a page appears in its AI features are the normal Search preview controls: nosnippet, data-nosnippet, max-snippet and noindex. Search Console also has a generative AI setting, on by default for every property, that lets an owner exclude the site’s links and content from AI Overviews, AI Mode and generative AI features in Discover. Google says that exclusion does not affect AI training.

Putting those pages together is our reading of Google’s documentation, since Google does not state the combination in one place: Google-Extended is a training and Gemini grounding decision, and the AI Overviews decision lives in Search. For a dealer that wants to be cited, leave the Search Console setting on and avoid nosnippet on pages you want quoted. More on Google AI Overviews for car dealers.

Do all bots obey robots.txt?

No. RFC 9309, the robots.txt standard, says its rules are not a form of access authorization, so compliance is up to each crawler. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Cloudflare notes that robots.txt compliance is voluntary.

The user-triggered bots are a special case for a reason: they fetch a page because a person asked for it. If a buyer pastes one of your vehicle pages into ChatGPT and asks whether the price is fair, the fetch that may follow is that buyer’s request. Blocking fetches like that mostly gets in the way of your own shoppers.

For the crawlers that honor robots.txt, the file does its job. For anything else, the control sits at the security layer, which is where Cloudflare’s AI bot settings and similar tools come in. Anthropic says its bots honor robots.txt and that blocking them by IP address may not work correctly, so for Claude, robots.txt is the right tool.

What robots.txt lines should a dealership use?

Use a robots.txt that names each AI search crawler and allows it, then adds training blocks only if the store has decided on them. The lines below are an example to adapt with your website vendor, never a file to paste over your current one, since that file may hold rules your site needs.

Keep your existing rulesStart from the current file. Rules that keep crawlers out of admin pages, internal search results or checkout flows usually belong there, and your vendor added them for a reason. Add the AI groups below them instead of starting over.
Allow the search and retrieval bots by nameAdd one group per crawler: a line reading User-agent: OAI-SearchBot followed by a line reading Allow: /, then the same two lines for Claude-SearchBot, Claude-User and PerplexityBot. With a named group, each of these crawlers follows its own rules instead of the catch-all group.
Decide on training, then write it downIf the store opts out of model training, add User-agent: GPTBot with Disallow: /, and the same for ClaudeBot and Applebot-Extended. Weigh Google-Extended on its own, since Google says it also controls grounding in Gemini Apps. Apple says Applebot-Extended does not crawl pages; it only controls training use. Applebot, the crawler behind Apple’s search features in Spotlight, Siri and Safari, is a separate name.
Check the catch-all groupLook at User-agent: * last. If it says Disallow: /, every crawler without its own group is shut out, which is why the named Allow groups in step 2 matter. Never add a blanket Disallow: / to settle a training worry.
Test after 24 hoursOpenAI says robots.txt changes take about 24 hours to be reflected, and Perplexity says up to 24 hours. Open yourstore.com/robots.txt the next day, confirm every group reads the way you wrote it, and date the check.

Is robots.txt the way to keep a page out of search?

No. Google says robots.txt mainly manages crawler traffic and is not a mechanism for keeping a page out of Google. To keep a page out of search results, use a noindex tag or password protection. OpenAI makes a similar point for ChatGPT Atlas, where a disallowed page can still show as a link and title.

This matters for pages a dealer genuinely wants hidden, such as a staff-only price sheet, a test landing page or an old campaign. Blocking them in robots.txt can leave the address findable with no description. A noindex tag, with the crawler allowed to read it, is the cleaner tool, and anything with private information belongs behind a login.

For everything a buyer should find, including inventory, the service menu, hours and directions, do the opposite: let the search crawlers in and make each page easy to quote. Getting those rules right, and keeping them right after every site change, is part of the monthly website work in our service.

Sources

Every bot description in this article comes from the company that runs the bot: OpenAI, Anthropic, Perplexity, Google and Apple, plus the robots.txt standard and Cloudflare’s documentation. Crawler names and rules change from time to time, so check the live page before editing your file.

Frequently asked questions

Will blocking GPTBot keep my dealership out of ChatGPT?

Not out of ChatGPT search. OpenAI describes GPTBot as its training crawler and treats blocking it as separate from OAI-SearchBot, the crawler that surfaces sites in ChatGPT search. A store can disallow GPTBot and stay eligible for ChatGPT search results, as long as OAI-SearchBot and OpenAI’s published IP ranges are allowed.

Can I block AI training but still allow AI search?

Yes. Name each crawler in robots.txt: allow OAI-SearchBot, Claude-SearchBot, Claude-User and PerplexityBot, and disallow GPTBot, ClaudeBot and Applebot-Extended if the store wants to opt out of training. Decide on Google-Extended separately, because Google says it also covers grounding in Gemini Apps. Avoid a blanket rule for every bot, which would block search along with training.

Does Applebot matter for a car dealership?

It can. Apple says Applebot data feeds search features across Spotlight, Siri and Safari, so keep Applebot allowed. If the store wants to opt out of Apple’s model training, disallow Applebot-Extended, which Apple says does not crawl pages itself.

How long do robots.txt changes take to work?

OpenAI says about 24 hours for OAI-SearchBot, and Perplexity says up to 24 hours for PerplexityBot. Check the live file the next day, then allow a few weeks before judging whether assistants have read your pages again, since crawling and answering run on their own schedules.

Check what your robots.txt lets in

The free scan reads your robots.txt and checks whether the security service in front of your site challenges automated visitors, then shows who ChatGPT and Claude name when local buyers ask. A person walks you through it in 20 minutes.

Get my free scan →

Free · No logins needed · Checked by a person · 20-minute walkthrough