Unblock the crawlers, then test live — robots.txt is not the report

If GPTBot or PerplexityBot cannot fetch your site, no amount of schema theatre will put you in an AI answer. Check that first. Then stop. A green setup scan is not the same as ChatGPT naming you when a buyer asks who to hire. Use the free scan to clear the floor. Use a live test for the sentence.

The check (five minutes)

1. Run the free site scan. Paste your URL. No account. It checks crawlers, homepage schema, and whether Bing has seen the domain. It does not call ChatGPT, Google AI, or Perplexity.
2. Or open https://yoursite.com/robots.txt yourself. Look for Disallow: / and for GPTBot, PerplexityBot, ClaudeBot, Google-Extended, OAI-SearchBot, Bingbot. The crawler-only check reads that file if you want a yes/no per bot.
3. If a bot you care about is blocked, fix that and re-scan. Do not rewrite the homepage yet.
4. If crawlers are allowed, ask ChatGPT, Google AI, and Perplexity: best [category] (add a city only if you are a local service). Write down who they name.

Optional: generate a starter llms.txt so there is a plain-language file at /llms.txt describing the product. It will not rescue a blocked crawler. It will not replace a live recommendation test.

What the free tools are for

Site scan. The starting check. Crawlers, schema, Bing index — one page. Still not “who did the model name.”

Crawler check. Fetchability only. Yes or no, per bot.

llms.txt creator. A draft file you can upload. Useful as a clear summary. Not a visibility score.

Index check / AI readiness. The same setup pieces, split out if you want one at a time.

Those tools are pre-checks. They are not the diagnostic.

What to do this week

1. Unblock any AI crawler you did not mean to block.
2. Put a 75–150 word answer at the top of the money page.
3. Re-run the three-engine buyer prompt. Screenshot it.
4. Only if you are fetchable and still unnamed: look at citations, entity, and the 12-step plan — not another robots pass.

If you want the live transcript — 20 prompts, ChatGPT, Google AI, Perplexity, who got named, top blockers, 30-day sequence — that is the one-time report after the free site scan is clean. One payment. About five minutes after intake.

Frequently asked questions

If crawlers are allowed, am I visible in AI search?

No. Allowed bots mean the page can be fetched. Visibility is whether ChatGPT, Google AI, or Perplexity named you on a buyer prompt. Fix robots first if you are blocked. Then test the sentence a customer hears.

Which bots should I check?

At minimum GPTBot, PerplexityBot, ClaudeBot, Google-Extended, and OAI-SearchBot / Bingbot if you care about ChatGPT Search. A Disallow: / or a bot-specific disallow is the first thing to fix.

What is the free site scan?

A no-account check of crawlers, homepage schema, and Bing index. It does not call ChatGPT, Google AI, or Perplexity. Those live tests are the paid report.

When should I pay for a diagnostic?

After the free site scan is clean and you still lose the recommendation prompt. A live three-engine test with who got named and a 12-step plan is the next job. Another robots scan is not.