Scoring methodology

The readiness score is not a vibe. Every point comes from a request we actually made to your domain, and every check below is the same code that produces your report — the weights on this page are read from the same source file the scorer uses, so this page cannot drift from your score.

Share this page X LinkedIn

How the number is produced

We fetch your homepage, follow up to 10 internal links, and fetch robots.txt, sitemaps and the agent file paths.
We hard-fail the scan if the domain redirects off-origin or serves a parking page — we would rather report nothing than score someone else's site.
Each of the 10 checks passes or fails on evidence. Passed weights are summed and divided by 112 total points, then rounded to a 0–100 score.
A check we could not test (for example a file behind a 403) is recorded as failed with the reason shown, never silently skipped.

Score bands

Agent-ready

75+ points

Agents can find, read and act on this site.

Partially ready

45+ points

The basics are there but agents are missing key signals.

Not ready

0+ points

Agents get little or nothing useful from this site.

The 10 checks

llms.txt present

16 / 112 points

How we test it: GET /llms.txt, falling back to /llms-full.txt. Must return 2xx, must not be HTML, and must not match the body this host returns for a random non-existent path.
Why it matters: A curated, plain-markdown map of your site is the cheapest way for a model to learn what you offer without crawling every page.

Reference: llms.txt proposal

agents.md instructions

12 / 112 points

How we test it: GET /agents.md, verified the same way as llms.txt.
Why it matters: States allowed and disallowed actions, rate limits and a contact, so an agent acting on your site behaves predictably.

Reference: AGENTS.md convention

agent.json manifest

12 / 112 points

How we test it: GET /agent.json, falling back to /.well-known/agent.json. Body must parse as JSON.
Why it matters: A machine-readable capability manifest lets an agent decide what it can do programmatically instead of guessing from HTML.

AI crawlers allowed in robots.txt

12 / 112 points

How we test it: Parse robots.txt into user-agent groups and check whether GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot, cohere-ai, DuckAssistBot, YouBot or Diffbot are disallowed at /.
Why it matters: If robots.txt blocks these agents, nothing else on the report matters — they never read a byte of your site.

Reference: robots.txt specification

Structured data (JSON-LD)

16 / 112 points

How we test it: Parse every <script type="application/ld+json"> block on the crawled pages, including @graph. Malformed JSON counts as absent. Passes when at least half the crawled pages carry valid JSON-LD.
Why it matters: JSON-LD is the only part of a page whose meaning is unambiguous — it turns prose into facts an agent can act on.

Reference: schema.org

Page metadata

12 / 112 points

How we test it: Every crawled page needs a title of 10+ characters, titles must be distinct across pages, and at least 70% need a meta description of 40+ characters. Homepage canonical is reported.
Why it matters: Titles and descriptions are what appear in an agent's short-list of candidate pages before it fetches anything.

Sitemap & robots

10 / 112 points

How we test it: GET /robots.txt, then /sitemap.xml, the Sitemap: directive in robots.txt, or /sitemap_index.xml. The sitemap must be XML containing <loc> entries; nested index files are followed one level.
Why it matters: A sitemap is how an agent enumerates your content instead of discovering it link by link.

Semantic HTML

8 / 112 points

How we test it: At least 70% of crawled pages must have exactly one H1 and no skipped heading levels. Presence of main/header/nav/footer/article landmarks is reported.
Why it matters: Heading hierarchy is how a parser segments a page into sections it can quote accurately.

Content in server-rendered HTML

6 / 112 points

How we test it: Strip scripts, styles and markup from the homepage response. Fails when visible text is under 400 characters and the HTML contains an empty #root, #app or #__next mount point.
Why it matters: Most AI crawlers do not execute JavaScript. If your content only appears after hydration, they see an empty page.

Public API or feed

8 / 112 points

How we test it: GET /openapi.json (must parse as JSON), /rss.xml and /feed.xml (must contain <rss> or <feed>).
Why it matters: A structured endpoint is the highest-fidelity path for an agent — no parsing, no ambiguity.

The robots.txt AI-crawler audit

This is worth 12 of the 112 points, and it is the one check that can invalidate every other one: if robots.txt disallows an AI crawler at /, that agent never fetches your llms.txt, your JSON-LD or anything else. We parse robots.txt into user-agent groups, resolve each bot below to its most specific group (falling back to User-agent: *), and treat Disallow: / or Disallow: /* as blocked unless a later Allow: / in the same group re-opens it. Comments are stripped and matching is case-insensitive, exactly as the specification requires.

User-agent	Operator	What it does	Cost of blocking it
GPTBot	OpenAI	Crawls pages to build OpenAI's training and retrieval corpus.	ChatGPT has no durable knowledge of your product beyond what it saw elsewhere.
OAI-SearchBot	OpenAI	Indexes pages for ChatGPT search results and citations.	You cannot be cited as a source in ChatGPT search answers.
ChatGPT-User	OpenAI	Fetches a page live when a user or agent follows a link during a conversation.	Agents acting for a real user cannot open your pages at all — the highest-intent traffic there is.
ClaudeBot	Anthropic	Crawls pages for Claude's index and training corpus.	Claude answers about your category are built without you.
Claude-Web	Anthropic	Legacy Claude web-browsing fetcher.	Older Claude browsing sessions cannot read your pages.
anthropic-ai	Anthropic	Legacy Anthropic crawler user-agent still honoured by many robots.txt files.	Blocked here usually means blocked by copy-pasted boilerplate rather than a decision.
PerplexityBot	Perplexity	Indexes pages for Perplexity's answer engine.	No Perplexity citations, which are one of the largest sources of AI referral clicks today.
Perplexity-User	Perplexity	Live fetch when a Perplexity user opens a result.	Users who click through from Perplexity hit a page the assistant could not read.
Google-Extended	Google	Controls use of your content in Gemini and AI Overviews (it does not affect Search ranking).	You are excluded from Gemini grounding and AI Overview summaries while classic Search is unaffected.
Applebot-Extended	Apple	Controls use of your content by Apple Intelligence.	Apple Intelligence features will not summarise or surface your content.
CCBot	Common Crawl	Builds the open Common Crawl archive that many models train on.	You disappear from the dataset a long tail of open models learn from.
Bytespider	ByteDance	Crawls pages for ByteDance / Doubao models.	No presence in ByteDance's assistants — relevant if you sell into APAC markets.
meta-externalagent	Meta	Crawls pages for Meta AI training and retrieval.	Meta AI answers omit your product.
Amazonbot	Amazon	Crawls pages for Alexa and Rufus-style shopping answers.	Amazon's assistants cannot quote your pages or prices.
cohere-ai	Cohere	Crawls pages for Cohere's retrieval and training corpus.	Enterprise assistants built on Cohere cannot ground answers on your site.
DuckAssistBot	DuckDuckGo	Fetches pages for DuckDuckGo's AI answers.	No DuckAssist summaries or citations.
YouBot	You.com	Indexes pages for You.com's answer engine.	You.com answers cite competitors instead.
Diffbot	Diffbot	Extracts structured entities that feed many commercial knowledge graphs.	Your company and product entities go missing from data products agents query.

What to do when bots come back blocked

Check whether the block is deliberate. A blanket User-agent: * plus Disallow: / — common on staging configs and copy-pasted templates — blocks all 18 of these at once, usually by accident. Your report flags that case separately.
Decide per operator, not globally. Blocking training crawlers (GPTBot, CCBot, meta-externalagent) while allowing the live fetchers and search indexers (ChatGPT-User, OAI-SearchBot, PerplexityBot, Perplexity-User, DuckAssistBot) keeps you citable and clickable without contributing to training corpora.
Add an explicit group per bot you want to allow. A specific User-agent group wins over *, so you do not have to loosen the wildcard rule that protects the rest of your site.
Remember what robots.txt cannot do: Google-Extended and Applebot-Extended only govern AI use — they never affect classic Google or Apple search ranking, so blocking them costs you AI visibility for no SEO gain.
Deploy robots.txt at your domain root, then re-run the scan. The check re-tests on every scan and scheduled re-crawl, and drift alerts fire if a future deploy blocks these agents again.

Allow every audited agent by appending this to /robots.txt:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: CCBot
Allow: /

User-agent: Bytespider
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: YouBot
Allow: /

User-agent: Diffbot
Allow: /

Sitemap: https://example.com/sitemap.xml

Limits we will not hide from you

A scan reads your homepage plus up to 10 internal links — enough to judge site-wide patterns, not an exhaustive audit of every page.
We fetch HTML as it is served. We do not execute JavaScript, on purpose: that is what most AI crawlers do, and it is how the "content in server-rendered HTML" check catches blind spots.
Weights are our editorial judgement of what agents rely on today, and will change as the conventions do. Changes are reflected here the moment they ship.

Run a free scan on your own site