20 August 2026
A crawl score is not what ChatGPT told your customer
Robots.txt and schema matter. They are not the sentence a buyer hears when they ask an AI who to hire. Here is the difference between a site scan and a live engine test.
A site scan that scores crawler access, schema, and speed is useful. It is also not what your customer was told. They asked ChatGPT, Google’s AI, or Perplexity who to hire. One of those systems named someone. If that someone was not you, the scorecard did not fail — the recommendation did. Those are different failures. Mixing them up wastes a week on markup while the answer stays the same.
The check (five minutes)
Run one prompt on three engines: best [category] in [city].
For each engine write down:
- Were you named?
- Who was named instead?
- Did it cite a URL you control?
Then open /robots.txt. If GPTBot, ClaudeBot, PerplexityBot, or Google-Extended are blocked, stop and fix that before you argue about content. A live test against a blocked site is noise.
What a scan is for
Scans catch the boring blockers: disallowed bots, missing Organization or LocalBusiness JSON-LD, no HTTPS, a homepage that is an empty shell until JavaScript runs. Those are real. They are also invisible to the buyer. Nobody asks ChatGPT for your schema validator result.
What a live test is for
ChatGPT, Google AI, and Perplexity do not share one retrieval stack. Google AI is still Google’s index plus a passage picker. ChatGPT leans on Bing and its own browsing. Perplexity retrieves on its own. A page can be “fine” on a crawl and still lose the recommendation on two of three engines.
The live test is the transcript: this prompt, this engine, this name. That is evidence you can hand to a developer. A letter grade is not.
Use both, in that order
- Unblock crawlers if they are blocked.
- Put a 75–150 word answer at the top of the money page.
- Re-run the three prompts. Keep screenshots.
- Only then chase directories, reviews, and extra pages.
If you want that transcript done for you — 20 prompts, three engines, who got named, top 3 blockers, 12 steps over 30 days — that is the one-time report this site sells. One payment. About five minutes after intake. Not another monthly dashboard.
Frequently asked questions
Should I ignore robots.txt and schema?
No. If GPTBot or Google-Extended cannot fetch you, no live test will save the page. Fix crawler access first. Then test what the engines actually say.
What is a live engine test?
The same buyer prompts run on ChatGPT, Google AI, and Perplexity. You record whether you were named, who was named instead, and whether any citation is a page you control.
Why would the score look fine and the answer still skip me?
A scan measures fetchability and markup. The answer is a retrieval choice. Weak entity data, no extractable first paragraph, or a competitor with denser third-party mentions can win the sentence while your crawl looks healthy.
How often should I re-run the prompts?
Once for a baseline, then after you ship the first two or three fixes. AI answers vary between runs. Treat it as a diagnostic, not a ranking tracker.