SEO Intel AI Visibility Playbooks · Signal 1 of 9

AI Crawler Access: robots.txt

AI search engines send their own crawlers to read your store. If your robots.txt file turns them away, your store cannot appear in their answers.

Run your free AI Visibility audit All 9 signals

Why it matters

Your robots.txt file is a short set of instructions that tells visiting crawlers what they may read. AI search engines such as ChatGPT, Claude, Perplexity and Gemini each send their own crawlers. If the file turns them away, they cannot read your pages, and your store cannot be found or quoted in their answers, however good everything else is.

Stores often block AI crawlers without knowing it: an old app, a theme setting or a file copied from another site can do it. It is one of the first things worth checking.

Want to know exactly what is wrong on your store? The free audit shows you where you stand today. The SEO Intel Skills Pack checks this across your whole store, tells you exactly what to fix, and fixes it with your approval. Get the pack →

Sources and references

Frequently asked questions

Can robots.txt stop AI search engines finding my store?
Yes. If your robots.txt file blocks their crawlers, AI search engines cannot read your pages and are unlikely to mention your store.
Will blocking AI crawlers affect my normal Google ranking?
Your normal Google ranking is handled separately, but blocking AI crawlers can keep your store out of AI answers, including some of Google's own AI features.
How do I know if my store is blocked?
Run the free audit on this site for a quick check. The SEO Intel Skills Pack checks your whole set-up and fixes it with your approval.

Skip the mistakes we made

This guide comes from a real problem on one of our own stores. Here's how to skip it.

1

See what's wrong

Run the free store audit. No account, no card. Built for Shopify; Wix stores get a quick check too.

Run the free audit
2

Fix it yourself with Claude

Skills Pack (Shopify, £14.99) checks and fixes an existing store. Store Builder (Shopify and Wix, £29.99) sets a new store up properly. Which one?

Skills Pack Store Builder
3

Learn how I did it

I cleared my own suspension without paying anyone. The full story, every mistake and every fix, is free to read.

Read how I did it