Free audit
SEO Intel AI Visibility Playbooks · Signal 1 of 9

AI Crawler Access: robots.txt

At 25 points, this is the single highest-scoring signal in the AI Readiness Score. A blocked robots.txt makes your entire store invisible to AI search engines regardless of how well everything else is optimised.

Run your free AI Visibility audit All 9 signals
25 pointsHighest-impact signal10-minute fix
25 points available
Highest-impact signal

What this signal measures

Your robots.txt file tells web crawlers what they are and are not allowed to access on your store. AI search engines operate their own dedicated crawlers: GPTBot and OAI-SearchBot for ChatGPT, ClaudeBot and Claude-SearchBot for Claude, PerplexityBot for Perplexity, Google-Extended for Google AI Overviews, and Anthropic-AI for Anthropic. All of these crawlers read robots.txt before fetching any page and obey its rules exactly.

At 25 points, the AI Crawler Access signal is the single highest-scoring check in the AI Readiness Score. A blocked robots.txt makes every other signal irrelevant: if the crawler cannot enter your store, it cannot read your descriptions, discover your FAQ, or verify your brand schema.

How the score is calculated

SEO Intel fetches your robots.txt at yourstore.com/robots.txt and walks it line by line. It checks for explicit Disallow: / rules for any of the 7 known AI crawlers, and also checks for a wildcard User-agent: * block with no per-bot AI override.

Common failure modes

Issue 1: Most common

Wildcard block with no AI override

A robots.txt containing User-agent: * followed by Disallow: / was originally set during store development (Shopify enables password protection by default) and was never updated when the store went live. This blocks all unrecognised crawlers. AI bots vary in how strictly they interpret this rule.

Issue 2

Explicit AI bot blocks added by SEO plugins

Some SEO management plugins maintain a crawler blocklist and add explicit Disallow: / entries for GPTBot or Google-Extended, sometimes at the merchant's request and sometimes by default. If GPTBot was added to the blocklist before the plugin was removed, the rule persists in robots.txt.liquid even after the plugin is gone.

Issue 3

Store still in password-protected mode

A store in Shopify password-protected mode returns a restricted robots.txt that blocks all crawlers. Neither AI search engines nor Google can index the store until the password is removed.

How to fix it

The fix has two parts: remove any blocking rules for AI crawlers, and optionally add explicit Allow rules to make access unambiguous.

Step 1. Visit yourstore.com/robots.txt in a browser. Read it. Look for any User-agent: GPTBot, User-agent: ClaudeBot, or User-agent: PerplexityBot lines followed by Disallow: /. Those lines are the problem.

Step 2. In Shopify Admin, go to Online Store → Themes → Edit code. Look for templates/robots.txt.liquid. Delete or comment out any Disallow: / line for AI crawlers.

Step 3. Add explicit Allow rules for each AI crawler above any wildcard block. Paste this before the User-agent: * section:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Anthropic-AI
Allow: /

Step 4. If your store is still password-protected, go to Online Store → Preferences → Password protection and disable the password. No crawler, AI or otherwise, can access a password-protected store.

Check your current robots.txt status in 60 seconds. Run the free AI Visibility audit and the report will show exactly which AI crawlers are blocked or allowed on your store. Also see the Free audit for Google-specific crawl and feed issues.

Sources and references

Read nextFAQPage Structured Data15 points

Frequently asked questions

What AI crawlers does robots.txt affect?
The robots.txt file affects GPTBot and OAI-SearchBot (ChatGPT), ClaudeBot and Claude-SearchBot (Claude), PerplexityBot (Perplexity), Google-Extended (Google AI Overviews), and Anthropic-AI. All of these crawlers read robots.txt before accessing any page and obey Disallow rules. Blocking even one of them means your store content cannot be used to answer queries on that platform.
Will blocking AI crawlers affect my Google search ranking?
Blocking AI-specific crawlers like GPTBot does not directly affect your Google organic search ranking, which is handled by Googlebot. However, blocking Google-Extended specifically does prevent your content from appearing in Google AI Overviews (formerly SGE). The risk of blocking AI crawlers is invisibility in AI-powered search, not traditional Google ranking.
How do I check what my robots.txt currently says?
Go to yourstore.com/robots.txt in a browser. The file is always publicly readable. You are looking for User-agent lines followed by Disallow: / lines. Any combination that names an AI crawler and has Disallow: / means that bot is blocked. The free SEO Intel AI Visibility audit checks this automatically and tells you exactly which crawlers are affected.