# The AI Readiness Index **Version 12.0 — specification** Effective 2026-10-18. Repository: [ten7/readinessindex-openstandard](https://github.com/ten7/readinessindex-openstandard) **License:** This document and the tests it defines are published under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). Copy it, adapt it, build your own scanner from it — for any purpose, free, forever — as long as you credit TEN7 as the creator of the AI Readiness Index and license what you build the same way. Scanners that compute a score from this document are separate software and are not part of the standard. Full terms at [openstandard.readinessindex.io/about](https://openstandard.readinessindex.io/about/#license). ## What this is This document defines what a website has to do to score 100 on the AI Readiness Index. It is written for two readers at once: the engineer building a scanner, and the consultant explaining a score to a client. Every test has an ID, a plain-English question, a detection method, and a scoring rubric. Every test also carries an evidence label and graded sources that say whether anything is documented using what it checks. If a test is not described here, the scanner should not score it. We built this because "is your site ready for AI" is currently an unanswerable question. Vendors assert things. Nobody measures. Our answer is a number you can reproduce, defend line by line, and re-run in six months to show what changed. The Index measures one thing: **can machines find, read, trust, and use this website?** It does not measure design quality, marketing effectiveness, or whether the content is any good. A beautiful site can score 12. A plain one can score 94. Four scored categories answer that question in order. **Reach:** can they get to it? **Read:** can they find your pages and read the words? **Understand:** do they know what it is? **Act:** can an agent do something? Only what has documented use is scored. Everything else is measured and shown in three groups beside the score (Posture, Extras and Measurement) and never moves it. ## How to read a score A result has three parts, and all three matter. **The score (0–100).** A single weighted number across four categories: Reach, Read, Understand and Act. **The gate status.** Two tests are gates. If a site blocks the search crawlers of AI assistants in `robots.txt` (RCH-1) or turns them away at the CDN or the firewall (RCH-2), `gated` is `true`: a separate fact reported beside the score, not a ceiling on it. The score still says how well the site is built; the gate says whether any of it is reachable today. A perfectly marked-up site that returns 403 to `Claude-SearchBot` is not badly built, it is invisible, and those are different problems with different fixes. Blending them into one capped number made a well-built gated site indistinguishable from a badly-built one; see the changelog for 4.0. Blocking a training crawler never gates: that is a rights choice, and PST-1 shows it without scoring it. **The automated coverage.** The percentage of applicable weight the scanner verified on its own. Some tests, such as whether the date a page shows is the date its markup says, can be fooled by a heuristic, and some facts cannot be seen from outside at all. Those are scored from attestation and flagged. A score of 82 with 91% automated coverage is a much stronger claim than 82 with 64%. Report all three together. A bare number invites arguments we don't need to have. Beside the score, every report shows three unscored groups: **Posture** (choices that are yours to make, which never lower readiness), **Extras** (good practice with no documented reader) and **Measurement** (how you track AI traffic). A site that does them is credited in words and is never marked down for leaving them out. ### Grade bands | Score | Grade | What it means | |---|---|---| | 90–100 | Exemplary | Machine-legible by design. Remaining work is optional or emerging. | | 75–89 | Strong | Solid fundamentals with specific, addressable gaps. | | 60–74 | Adequate | Findable and readable, but leaving significant value unclaimed. | | 40–59 | Weak | Partially legible. Real content is not reaching answer engines. | | 0–39 | Not ready | Structural problems. Start with retrievability before anything else. | | Any | **Gated** | A separate fact, not a grade. RCH-1 or RCH-2 failed: the search crawlers of AI assistants cannot reach the site, so none of the score above is reaching one today. Before 4.0 this capped the total at 40, which blended two facts into a number comparable to neither. | `grade` is always the band the real, uncapped `total` falls in — a gate failure never changes it. `gated` is the separate boolean the report checks to decide whether to lead with the access problem, so it can say "Strong at 81, and gated — none of that is reaching an assistant today" rather than either hiding the score behind the word Gated or letting it stand alone with no warning attached. ## The scoring model ### Bands Every check scores on a 0–4 band. The check's contribution is `weight × (band ÷ 4)`. Unless a test says otherwise, use the **default coverage rubric** against the sampled page set: | Band | Condition | |---|---| | 4 | Present and correct on ≥95% of applicable sampled pages | | 3 | Present and correct on 75–94% | | 2 | Present on 40–74%, or present everywhere but with correctness errors | | 1 | Present on 1–39% | | 0 | Absent, or present but broken everywhere | Coverage percentages are weighted, not raw page counts. Each money page (see Sampling, step 4) counts as **2** in both the numerator and the denominator; every other sampled page counts as 1. Ten money pages in a 250-URL sample therefore score as a 260-page sample. A failure on `/apply` costs twice what the same failure on a 2019 press release costs, which is the reason the client names those URLs in the first place. Tests that are inherently site-wide and binary (a file exists, a header is set) define their own bands in the test entry. ### Points | # | Category | Points | Tests | |---|---|---|---| | 1 | Reach | 25 | 4 | | 2 | Read | 35 | 19 | | 3 | Understand | 25 | 11 | | 4 | Act | 15 | 4 | | | **Total** | **100** | **38** | Beside the hundred, and counted separately because they are not part of it: 34 unscored tests in three groups. **Posture** has 6, **Extras** has 20 and **Measurement** has 8. A report shows all 72 tests. The score is made of 38. Weight follows three things: how badly a failure hurts, how strong the evidence is that it matters, and how many assistants it affects. - **Reach is 25 and Read is 35, so the two together are 60.** These are the categories where the evidence comes from the companies that run the assistants, and the only failures that make a site invisible. Reach holds the four tests that decide whether an AI system can get to a site at all. Read holds the tests of whether it can find the pages and read the words, which includes the crawl housekeeping (sitemaps, redirects, dead pages, filters, pagination, file rules) that used to sit beside Reach in Index 11. - **Understand is 25 and Act is 15.** Understand is documented for Google and Bing rather than for AI products, and it is led by dates (UND-6). Act has real but narrow evidence (browser agents read page structure and stop at CAPTCHAs), so it is four tests: ACT-1 and ACT-4, which both rest on the accessibility tree Anthropic documents its browser tool reading, carry five points each, ACT-2 carries three and ACT-3 two. Before marker Y, Reach and Read carried 75 points and Act carried five. - **Seven tests are labeled Practice and carry one point each** (RED-13 to RED-19). They are sound practice for people, and nothing documents an AI product using them. - **Rights are not scored.** A site's page settings decide whether a page can be quoted, so RCH-4 is scored in Reach. What a site permits AI to do with its content, and what it licenses, is a choice. PST-1 to PST-5 show it and never score it. - **Google's AI-control setting is not scored.** A scan cannot see it, and a number should not rest on something a scan cannot check. PST-6 asks the owner to confirm it. Posture, Extras and Measurement carry none of the 100 points — by design, not omission. Scoring a practice nobody has a documented use for subtracts the same points from every site, which moves nobody relative to anybody. They are still run and still reported. ### Applicability and N/A Each test declares an applicability rule: - **`always`** — scored on every site. - **`conditional`** — scored only if a precondition holds. State the precondition in the check. - **`vertical`** — scored only for a declared client vertical. Sixteen are published; `other` is one of them and carries no expected type set, so a client who tells us their industry is not on the list is not marked against a standard that does not exist for them. A conditional test states its precondition in one of two forms, and never in both: - **A named site fact** — a single named predicate over collected evidence, such as `sample_contains_tables` or `site_has_search`. Predicates are code, not expressions in a config file, so each one is unit-tested and each one names itself in the N/A reason an auditor reads. - **A dependency on another check** — written as `requires: RED-10 ≥ 2`. A dependency is satisfied when the named check scores **band 2 or higher**. Below band 2 the foundation is broken, and measuring what sits on top of it produces a finding nobody can act on: telling a client their Markdown copy is correctly advertised when the copy itself is broken is noise. A dependency that is itself N/A cascades N/A to everything that depends on it. Two tests carry a dependency: RED-11 → RED-10 and MSR-4 → MSR-3. #### Facts the scanner cannot observe Six tests are conditional on a business fact no crawler can determine: PST-3 (licensable content), EXT-3 (the site runs its own agents), EXT-8 (health, legal and financial content), EXT-9 (research and policy publishing), EXT-10 (original asset production) and MSR-1 (named critical flows). UND-2 needs a declared vertical. UND-9 needs to know whether the site has a paywall, which a crawler can often see and an owner's statement settles. These come from an engagement profile supplied per client — never from inference. The scanner does not sniff a site for medical keywords and decide it is YMYL. A guess inside a number we ask a client to defend line by line is worse than saying we have not checked. But not checking is not the same as not scoring, and from 1.0 to 9.x the Index treated them as the same thing. #### Every test is scored (10.0) **On any run a person is shown, every scored test comes back with a band.** The denominator is the full 100 points, on every site, every time. The unscored groups report a state and never enter the arithmetic. Applicability resolution still runs, and still decides exactly what it always decided: whether this test has anything to find on this site. What changed in 10.0 is only what happens next. A verdict that used to end the test's participation in the run now resolves to a band, in one of three ways: | What resolution found | Band | What the report says | |---|---|---| | The site has none of the thing the test is about — a precondition that evaluated false, a vertical the site is not in, a dependency whose parent had nothing to find | **4** | *There is none of this on the site, so there is nothing to fix.* | | The answer is real but invisible from outside — an undeclared profile fact, a manual test with no attestation, or evidence we read and could not settle | **2**, held | *We cannot see this from outside your site. Confirm it and this scores in full.* | | The test this one depends on is not passing | **0** | *The test this one builds on is not in place yet.* | The middle row is a **ceiling**, not an earned band, and uses exactly the mechanism `unattested_ceiling` already used: the band is held at 2, `band_capped_at` and `cap_reason` record why, and `earnable_points` falls accordingly. A held test can never read as a pass, and confirming it can always move the number. Points above a ceiling are also excluded from the backlog: a test sitting at its ceiling has nothing on the site to fix, and a list headed *what to fix* must not open with work nobody can collect. Every one of these carries its `unscorable_reason` in the result file and an evidence row naming what was looked for, what came back, and what it scored — the same audit trail as any other test. An auditor can still see exactly what we chose not to measure; it is now a fact about a scored row rather than a missing one. **Why.** A test with no band was invisible to the arithmetic *and* to the reader. A self-serve visitor declares nothing, so more than half the Index left the run — a third of it purely for want of an engagement profile — and the score that came back was a percentage of a denominator that silently changed from site to site. Two sites could not be compared, and a client reading their own report found rows that gave them neither a result nor anything to do. Scoring every test costs the Index nothing it was actually measuring and gives back the property its name claims: a hundred points, all of them accounted for. **What this is not.** It is not scoring an N/A as zero — penalizing a museum for having no `Physician` markup is the mistake this Index was built to avoid, and the first row above is what prevents it. Nor is it paying a site for our own blind spots: anything we could not see is held at 2, never granted, which is the same dishonesty pointing the other way. **Two runs are exempt.** A **Reach scan** looks at four tests and reports points out of 25, and says so; scoring the other thirty-four would hand it points for work it never did and move its ceiling from site to site. The **census** exists to report what share of sites do a given thing, so a test it could not judge has to stay out of that tally rather than be counted as a site with nothing to fix. Both run with `score_everything` off, and neither is a report a client is handed. ``` score = (Σ earned_points ÷ Σ applicable_weight) × 100 ``` ### Resolving external identifiers UND-10 asks whether a site's `sameAs` links point at real authority records. Classifying a URL as a tier-1 identifier needs no network call; confirming it still resolves does, and those hosts belong to somebody else — ORCID, Wikidata, a registry, a university directory. Requesting them tells a third party which site is being audited. That is a real disclosure, and it is why resolution is a deliberate choice rather than a default in the reference scanner's CLI. A scan that does not resolve them still grades UND-10 on the links as published, at medium confidence and saying so, because a scored test must be able to earn full marks: the links are classified, not confirmed. Two rules apply wherever it is turned on: - **A third party's availability is never a fact about the client's site.** A timeout, a DNS failure or a rate limit against ORCID records that one link as unconfirmed. It must not fail the scan, and it must not score the client down. - **Say that it happened.** The evidence records which external hosts were contacted, so a client can see what their audit disclosed and to whom. ### Rounding, and the full accounting **Every points figure shown to a reader is a whole number. Every points figure used in the arithmetic is not.** A test contributes `weight × band ÷ 4`. No weight in the Index divides by four, so a weight-2 test at band 3 contributes exactly 1.5 points and a weight-5 test at band 2 contributes exactly 2.5. Reports print those as 2 and 3. This is deliberate. The Index exists to be defended out loud, line by line, in a room with the people who have to do the work. Nobody argues about three quarters of a point, and a column of figures like `2.25` reads as a spreadsheet artifact rather than a judgment somebody made. Whole numbers are the readable choice. They are also, in aggregate, wrong — and saying so is not optional: - **A run's printed rows do not sum to its printed score.** On a typical run roughly a third of the rows round up or down. Adding the printed column can land ten points or more away from the printed total. - **The score is never computed from the rounded rows.** It is computed from the exact contributions, summed at full precision, divided by the total weight, and rounded exactly once, at the end. Both numbers are correct. They answer different questions, and only one of them is the score. **Every result therefore carries a full accounting**, reachable from the foot of the report and from the stored result file: every test, its weight, its band, the arithmetic `weight × band ÷ 4` written out, its exact contribution, the figure the report printed instead, and the final division. Nothing on that page is rounded except where it says so. It recomputes the score from weight and band rather than reprinting the stored figure, so a disagreement between the accounting and the report would be visible rather than silent. An implementer of this specification must do the same. Print whole numbers if you like, but publish the arithmetic, and never compute a score from figures you have already rounded. *The one real fix — re-weighting every test to a multiple of four, so a 400-point Index lands every band on a whole number — is not made here. It would be a second non-comparable version boundary bought for a cosmetic gain, and the accounting already makes the arithmetic checkable.* ### Gates Two tests are gates: **RCH-1** and **RCH-2**. If either scores 0 or 1, set `gated: true` and record which one in `gate_failures`. Nothing else changes: `total` is computed exactly as in *Scoring* above, from every applicable test including the gate tests themselves, and `grade` is the band that real `total` falls in. RCH-1 gates on the search crawlers of AI assistants, and a training crawler never counts toward it. RCH-2 gates on the edge: a firewall that refuses a search crawler removes the site from that assistant whatever `robots.txt` says. Before 4.0, a gate failure capped the final score at 40 (`total = min(ungated_score, 40)`) and forced `grade` to `Gated`. That produced one number standing for two different facts — how well the site is built, and whether any of it is reachable today — and made them impossible to tell apart: a gated site with excellent structure scored identically to a gated site with none. Reporting `gated` and `total` as two separate facts says both; capping said neither. ### Verification mode Every test declares how its band was determined: - **`auto`** — the scanner determined the band from evidence it collected. - **`auto-partial`** — the scanner produced a strong signal, but a human must confirm. Heuristics that can be fooled live here. - **`manual`** — not observable from outside. **No scored test uses this mode.** Every point in the hundred is something the scanner can see for itself; what cannot be observed is shown in Posture or Measurement instead. The mode remains defined because PST-6 and MSR-2 to MSR-6 use it. Automated coverage counts `auto-partial` at half weight, because a strong signal awaiting human confirmation is genuinely half-verified: ``` automated_coverage = (Σ auto_weight + 0.5 × Σ auto_partial_weight) ÷ Σ applicable_weight ``` The half is not arbitrary. Counting `auto-partial` at zero puts a ceiling of 0.92 on every possible run, because the always-applicable `auto-partial` tests (RED-3, RED-18 and UND-6) are 8 points that never leave the denominator. Counting it in full reports 1.00 for a site nobody has confirmed anything about, and erases the distinction the mode exists to draw. Never let a `manual` check silently default to 4. An unanswered attestation scores `null` and drops out of the denominator, and the result records it as unverified. #### Attestation against an `auto-partial` check Three `auto-partial` tests have a top band that asks something no scanner can see. This document says so in each case, and each has a different observable ceiling because their band tables differ: | Test | Unattested ceiling | What the attestation confirms | |---|---|---| | EXT-12 (IndexNow) | 3 | That IndexNow submissions fire on publish, not on a schedule | | EXT-6 (generated markup) | 3 | That JSON-LD comes from mapped fields, confirmed with the build team | | MSR-7 (search consoles) | 3 | That sitemaps are submitted and alerts and AI reports reach a named person | Each of these three now has a band 3 that describes exactly what a scanner can see, so the attestation lifts 3 to 4 rather than jumping a gap. A test whose band table skips a value cannot express *mostly right*, and *mostly right* is the state most real sites are in. An attestation may **raise the ceiling** on these three. It never sets a band and never lowers one: the scanner scores what it observed, and a supplied attestation permits that observed band to rise as far as the evidence supports. The result records which points came from a claim rather than a probe, and the report prints them as attested. Without an attestation the test scores at its observable ceiling and the report says what would be needed to go higher. A client who did the work can prove it; a client who says nothing earns nothing. Every other `auto-partial` test is fully observable. It carries that mode because the heuristic can be fooled — EXT-3 reads a key directory against what a draft requires, EXT-4 calls a live tool — not because it needs a client to vouch for it. #### Tests that need a previous run Two tests are defined against change over time: RED-3 (is `lastmod` a real signal or deploy noise) and UND-6 (do dates move only when content moves). On a first run there is nothing to compare against. Score the observable half, **cap the band at 3**, set `confidence: medium`, and record `stability_unconfirmed` in the evidence. UND-6 already words its band 3 this way; RED-3 follows the same rule. Supplying a previous run directory as a baseline lifts the cap and permits band 4. Every ceiling — this one, the attestation ceiling above, and a test's own declaration that it could not see further — records the reason it applied. A client told a number was held below the evidence is owed the sentence explaining what would lift it. This makes the re-audit a first-class input to the score rather than a report-writing exercise, and it gives the client a visible, earnable reason the number rises next quarter. ## How the audit runs ### Sampling Most tests score against a sampled page set, not the whole site. Build the sample like this: 1. **Always include:** the homepage, `/robots.txt`, `/sitemap.xml` (and every child sitemap), `/llms.txt`, `/.well-known/` probes, and every URL in the site's primary navigation. 2. **Stratify by content type.** Use the sitemap index, URL patterns, or a declared content-type map. Take up to 10 URLs per type. 3. **Add depth.** Include at least 5 URLs three or more levels deep, to catch templates the homepage never exercises. 4. **Add the money pages.** The client names up to 10 URLs that matter most — programs, services, donate, apply, contact. These are weighted double in coverage math. 5. **Cap the crawl.** Default 250 URLs. Configurable. Record the actual count. 6. **Read only HTML as a page.** A sampled address that answers with a PDF, an image or another document that is not HTML is recorded (its status, type and headers, so RED-5 and RED-18 still see it) and is not parsed, rendered or counted as a page in any page-level test. If every sampled address is such a document, they are read as before rather than scoring nothing. Minimum viable sample is 25 URLs. Mark the run `low_confidence` only when the sample is under 25 **and** more URLs were discovered than were sampled. A twelve-page nonprofit site that was crawled in full is a complete measurement, not a weak one. #### The representation sub-sample Three tests want an extra request against every sampled URL: RED-10 (the Markdown copy a page advertises, or `{path}.md`), RED-12 (`Accept: text/markdown`, and cache behavior) and RED-5 (`X-Robots-Tag` on non-HTML). At the default 250-URL cap and one request per second, that is over 600 requests and roughly ten minutes of politeness delay stacked on top of the base crawl, against infrastructure that often belongs to a hospital. Those three tests run against a **representation sub-sample**: 25 URLs by default, drawn from the main sample, always including every money page. The size is configurable and the selected URLs are recorded in the run metadata, so the sub-sample is reproducible across audits. Twenty-five pages resolves a rubric whose finest distinction is 75% against 95%. ### Fetching Fetch every sampled URL twice: - **Raw** — plain HTTP GET, no JavaScript, with a declared user-agent. - **Rendered** — a headless browser with JavaScript enabled. The difference between those two responses is the entire evidence base for RED-1 and RED-2, which together carry 15 points. Do not skip the second fetch. The Reach scan is the exception, and says so: it uses no browser, because none of its four tests needs one. Separately, probe a small set of URLs with each search crawler's user-agent to feed RCH-2. There is no other way to measure what the edge does: `robots.txt` states an intention, a CDN rule states a fact, and the two disagree often enough that a readiness score which skipped the question would be worth less than the awkwardness of asking it. Say plainly what that means, because it is the part an implementer will be asked about. **The wire-level `User-Agent` on those requests carries another company's product token** — `OAI-SearchBot`, `Claude-SearchBot`, and the others in the table below. Three constraints make that measurement rather than impersonation, and an implementation that drops any of them is doing something else: 1. **The requests still identify you.** Send your own `From` header and an `X-Audit-Operator` header naming your crawler on every probe. An operator inspecting the whole request, rather than the user-agent alone, can see who it was. 2. **Probe only URLs you were already allowed to fetch.** The probe runs over the sample, and the sample is filtered by `robots.txt` first. A probe must never reach, under somebody else's name, a page you declined to reach under your own. 3. **Publish the behavior before you do it**, at the URL in your own user-agent string, alongside how to opt out. Disclosure volunteered reads differently from disclosure discovered. #### The agent classes The Index groups the agents it names four ways. RCH-1 evaluates the search crawlers against `robots.txt` and the gate turns on them. RCH-2 probes the ones that actually issue requests. PST-1 shows the rest and never scores them. | Class | Agents | RCH-1 | RCH-2 probe | Shown in | |---|---|---|---|---| | Search and answer crawlers | `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Googlebot`, `bingbot`, `Applebot`, `MistralAI-Index`, `meta-webindexer`, `Amzn-SearchBot` | yes | yes | RCH-1, RCH-2 | | Fetchers that act on a person's request | `ChatGPT-User`, `Claude-User`, `Perplexity-User` | no | yes, where the operator publishes a user-agent string | PST-1 | | Training and corpus | `GPTBot`, `ClaudeBot`, `CCBot`, `Bytespider` | no | shown, never scored | PST-1 | | Directive-only | `Google-Extended`, `Applebot-Extended` | no | **no** | PST-1 | `Googlebot` and `bingbot` are in RCH-1 because the AI answers of Google and Microsoft are built on their indexes. `Applebot` follows the `Googlebot` group when `robots.txt` does not name it, and the verdict says which group it used. A fetcher that acts on a person's request is not held to `robots.txt` by its operator's own documentation in several cases, which is why it is listed in PST-1 and not gated on. `Google-Extended` and `Applebot-Extended` are not crawlers. They are opt-out tokens that Google and Apple honor in `robots.txt`; no request ever arrives carrying either string. Blocking them is a rights decision with no fetch behavior attached, which is why PST-1 lists them and RCH-2 cannot probe them. A user-agent matrix that lists them is describing a crawler that does not exist. xAI publishes no crawler documentation and no list of its addresses. The names `GrokBot`, `xAI-Grok`, `Grok-DeepSearch` and `xAI-SearchBot` are known only from third-party directories, so PST-1 reports whether `robots.txt` names them, says that, and scores nothing. #### The Reach scan The free scan runs RCH-1 to RCH-4: one page, `robots.txt` and the edge probe, with no browser and no sign-in. It reports points out of 25, and a site can score 25 of 25. A Reach score is not a score out of 100 and is never compared with one. The posture panel (including the prompt in PST-6), `llms.txt` and the other unscored items are shown from the same fetches, so the free scan reads nothing the full scan does not. ### Politeness Respect `Crawl-delay`. Default to 1 request per second with 4 concurrent workers. Back off on 429 and 503, honoring `Retry-After` where it is given, and abandon a host that keeps refusing rather than retrying into a wall. Cache every response to disk so re-scoring never re-crawls. ### Consent Two different questions, and conflating them is how a scanner ends up doing something indefensible. **Whose site may you scan?** A client audit runs with the client's agreement, recorded before the crawl. A survey of a public population — see *Populations*, below — runs without asking each site first, on the same footing as any other published measurement of the public web. What is not acceptable in either case is scanning under a consent you did not obtain. Record which basis a run used, in the run itself. **What may you fetch once you are there?** `robots.txt`, and specifically the group addressed to your own crawler. Name your crawler in your user-agent string, publish a page at the URL that string points to, and stop when a site names you and disallows you. A scanner may reasonably decide that a blanket `User-agent: *` disallow does not address it — a `*` rule is aimed at the anonymous scrape, and a named crawler with a published policy, a rate limit and a working opt-out is not that. If you make that choice, **publish it on your crawler's own page** and honor `Crawl-delay` under `*` regardless, because a rate limit is not an exclusion. What is not defensible is making the choice quietly. ### Populations A survey scores a defined population and publishes the aggregate. Two rules keep it a measurement rather than an accusation. **Cite the population.** Record the source list, its publication date and its URL. "How did you pick these" is the first question anyone asks, and the answer has to be a citation rather than a judgment call. Use sources you may lawfully redistribute, and claim no ownership over somebody else's list. **Publish the version and the date.** Every result records which version of this Index it was scored against and the date that version took effect. Without both, an old number silently starts meaning something new the next time the Index changes. ### Reproducibility Store the raw evidence — headers, HTML, extracted JSON-LD, screenshots — in a run directory keyed by timestamp. A score you cannot reproduce next quarter is an opinion, not a measurement. ## Result schema The scanner writes one `result.json` per run. Everything else (Markdown report, HTML dashboard, remediation backlog) renders from this file. ```json { "schema_version": "1.0", "index_version": "12.0", "site": { "name": "Example University", "base_url": "https://example.edu", "vertical": "higher-ed", "cms": "drupal-10" }, "run": { "id": "2026-11-03T20-14-33Z", "started_at": "2026-11-03T20:14:33Z", "finished_at": "2026-11-03T20:41:02Z", "scanner_version": "0.1.0", "urls_sampled": 187, "representation_sample": 25, "low_confidence": false, "baseline_run_id": null, "extraction_method": "readability-density", "extraction_version": "1" }, "score": { "total": 61.4, "gated": false, "gate_failures": [], "grade": "Adequate", "applicable_weight": 94, "earned_points": 58, "earnable_points": 88, "automated_coverage": 0.87 }, "dimensions": [ { "id": 3, "name": "Understand", "weight": 18, "applicable_weight": 18, "earned_points": 9, "score_pct": 50.0 } ], "checks": [ { "id": "UND-4", "title": "Does each article show who wrote it, with a page about the author?", "dimension": 3, "weight": 2, "applicability": "always", "applicable": true, "na_reason": null, "verification": "auto", "band": 2, "earned_points": 1, "confidence": "high", "summary": "3 of 7 distinct authors have a page or profile a machine can find.", "evidence": [ { "url": "https://example.edu/news/new-lab", "author": "A. Rivera", "profile_found": false } ], "remediation": { "severity": "major", "effort": "medium", "summary": "Give each author a page.", "detail": "Four recurring authors are named in bylines with nothing behind the name..." } } ], "remediation_backlog": [ { "rank": 1, "check": "RED-1", "points_available": 11, "severity": "critical", "effort": "large" } ] } ``` #### Schema notes - The `dimensions` array and each test's `dimension` hold the four categories: 1 Reach, 2 Read, 3 Understand, 4 Act. The names are kept so a stored result from an earlier Index still loads. - A test that is not scored (Posture, Extras and Measurement) carries `weight` 0 and no `dimension`, and never enters `score`, `applicable_weight` or `earnable_points`. - `site.cms` is best-effort, from the `generator` meta tag and response headers. It is nullable and nothing scores against it. - A category with no applicable tests reports `score_pct: null`, not `0`. Zero means measured and failed. - **Rounding happens once, at the end.** All arithmetic runs unrounded; the values written to this file are rounded to one decimal for display. The per-check `earned_points` therefore will not sum exactly to `score.earned_points`, and neither is wrong. Do not "fix" this by rounding intermediates — that reintroduces the drift the single rounding step exists to prevent. - `extraction_method` and `extraction_version` record how main-content text was isolated. When extraction improves, old scores stay interpretable because the method that produced them is on the record. ### Remediation ranking Sort the backlog by **points available** (`weight − earned_points`), then by effort ascending, then by check ID ascending. The last key is not cosmetic: two runs of the same evidence must produce the same backlog in the same order, or the re-audit comparison is unreadable. A check appears in the backlog only if it is applicable, has a band, and has points available above zero. N/A checks, unattested `manual` checks, and checks already at band 4 are not remediation items. Use three severities — `critical`, `major`, `minor` — and three effort sizes — `small`, `medium`, `large`. Effort is a property of the check, defined in this document. Severity is derived: gate failures and any check scoring **0 or 1** with weight ≥3 are `critical`; band ≤2 with weight ≥2 is `major`; everything else is `minor`. Band 1 on a heavy check is a critical finding. A site whose program pages ship 22% of their content scores RED-1 at band 1, not 0, and calling that `major` alongside a missing caption on a table understates it to the person deciding what to fund. The client-facing output is not the score. It is the ordered list of what to fix and what each fix is worth. ### Who fixes it Fixing a failed test is work, and different tests need different people: a firewall rule, a template, a paragraph on an About page, a decision about what the organization permits. Each entry therefore names who typically does the work in a **Fixed by** bullet, so a reader of a result knows whose desk it lands on. There are six roles. An entry lists one to three of them, **in order: the first is who typically does or drives the work, and the others are whose input or hands it also needs.** A decision an entry depends on, such as a licensing policy, is made by the **Site owner** and appears in that entry's Fix text as well. | Role | Who they typically are | What they do for the Index | |---|---|---| | Site owner | Leadership, the communications head, counsel | Decide what the organization permits, licenses and says in public. | | Content team | Editors, content strategists, communications and web staff | Write and maintain what pages and media say: facts, FAQs, alt text, authors, reviewers, citations. | | Search specialist | SEO and digital marketing, analytics | Own crawl and index signals, robots.txt, redirect and sitemap strategy, the search consoles, and the measurement of AI referrals. | | Front-end developer | Themer, UI engineer, accessibility developer | Own templates and markup: HTML structure, headings, tables, forms, and what renders without JavaScript. | | Back-end developer | CMS developer, platform engineer, module developer | Own the data model and what the site generates: structured data, sitemaps, feeds, redirects, APIs and Markdown. | | Operations | DevOps, infrastructure and security engineers, the CDN administrator | Own the edge and the servers: CDN and firewall rules, caching, headers, bot verification, logs and signing keys. | This is guidance, not a requirement, and it does not touch the score. Organizations divide the work differently: at a small nonprofit one person holds three of these roles, and at a university each is a team. The roles say what kind of skill the fix needs, not how many people that takes. A report that groups the work by role uses each entry's first role, so the points in the groups add up to the points available. ## What counts as evidence Every test carries one **evidence label**, and every source it cites carries a **grade**. Together they say how far a reader should lean on a test, and they decide whether it carries points. ### Evidence labels | Label | Meaning | |---|---| | **AI product** | A named AI product or agent is documented using it. The operator's own documentation says so. | | **Search index** | Documented for Google or Bing, not for an AI product. Google's own guide says no special markup is needed for its AI features, so this is weaker than it looks. | | **Standard only** | A published standard or draft with no documented consumer. | | **None found** | No source found either way. Absence of evidence is not evidence of absence, and the test says so. | | **Practice** | Sound practice with no documented consumer. It is good for people, and no AI company documents using it. | | **Scanner-listed** | A published scanner checks it (Cloudflare's Agent Readiness score, Lighthouse's agentic browsing category) and no AI product is documented using it. A scanner measures; it is not a reader. Shown and never scored. | A label may carry a short parenthetical that narrows it, such as (coding agents only). The parenthetical is part of the claim: a test credited for coding agents is not credited for search crawlers. ### Source grades | Grade | Meaning | |---|---| | **A** | Operator or standards-body documentation, read directly from the source. | | **B** | An independent measurement or study, or a vendor's own data. | | **C** | A single blog or a third-party directory. | A scored test must not rest on a grade C source alone. A source whose address or date could not be confirmed is marked so in the test's **Sources** line. ### Policy - **A test is scored only if it has documented use.** That means the AI product or Search index label, or the Practice label, which is scored at one point and shown as Practice. A test labeled Standard only, Scanner-listed or None found is measured and shown in Posture, Extras or Measurement, and is worth no points. A site that does it is credited in words and is never marked down for not doing it. - **Every test carries a review date six months out.** The date is on the test's **Review by** line: 2027-04-18 for the tests as published in 12.0. A test is also re-read when a source it cites moves, changes or disappears, and it is changed if the reading changed. - **A label that moves a test into or out of the score is a change to the score,** and follows *Versioning and change control*. A corrected citation or a reworded question is a minor change. --- # The tests Each entry gives: the question the test asks in plain language, its points, applicability, verification mode, effort, who typically fixes it, the evidence label, what the test measures, why it matters with a source that is not us, the full list of graded sources, how to detect it, what evidence to keep, the band rule when it differs from the default coverage rubric, how to fix it, what changed from Index 11, which Index 11 test it replaces, and when it is next reviewed. The **Question** and **Why this matters** bullets are written for somebody who has never read this document, in plain words with the technical terms left to **Detect**. **Why this matters** is a plain description of the test, then a link to the documentation of a standards body, a search engine or an AI company that backs it, then one sentence on why that source is worth citing. A scanner or a report that explains a test to a client uses those words rather than its own, so every explanation of a test says the same thing. **Evidence** is the label defined in *What counts as evidence*, and **Sources** is every citation behind it with its date and grade. **Evidence kept** is what the scanner records for the run. A test that replaces one or more Index 11 tests says so in **Replaces**. The full map is in *The crosswalk from Index 11*. ## #1 · Reach · 4 tests · 25 pts **Points 25** These four tests can each be answered from a single page and the site's front-door files, `robots.txt` and the edge, which is what makes the free Reach scan possible: it reads one page, runs exactly these four, uses no browser and reports points out of 25. Every test in the groups that follow needs more than one page, or a render, or a fetch of something beside the page, and runs only in a full scan. ### RCH-1 — Can the AI tools that search the web read your site? - **Question:** Can the AI tools that search the web read your site? - **Points:** 8 - **Gate** - **Applicability:** always - **Verification:** auto - **Effort:** small - **Fixed by:** Search specialist, Site owner - **Evidence:** AI product - **Measures:** We read your robots.txt file, which tells crawlers what they may read, for the search crawlers behind ChatGPT, Claude, Perplexity, Google, Bing, Apple, Meta, Amazon and Mistral. If one is blocked, that assistant cannot cite you. - **Why this matters:** Your robots.txt tells crawlers what they may read. If it shuts out the search crawler behind an assistant, that assistant cannot cite you. [OpenAI, crawler overview](https://developers.openai.com/api/docs/bots): OpenAI names its own crawlers there, so it is the place to confirm which names robots.txt has to address. - **Sources:** [OpenAI, crawler overview](https://developers.openai.com/api/docs/bots), undated (A); [Anthropic Help Center, crawler controls](https://support.claude.com/en/articles/8896518), 2026-04-07 (A); [Perplexity, crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), undated (A); [Meta, web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers), undated (A); [Apple, About Applebot](https://support.apple.com/en-us/119829), 2026-09-04 (A) - **Detect:** Parse `robots.txt`. Evaluate each of the nine search crawlers (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Googlebot`, `bingbot`, `Applebot`, `MistralAI-Index`, `meta-webindexer`, `Amzn-SearchBot`) against the rule set, for `/` and for three sampled content URLs. Resolve `User-agent: *` fallbacks and the longest-match group rule, and let a crawler inherit the group it falls back to: Applebot follows the `Googlebot` group when `robots.txt` does not name it. Training crawlers and the fetchers that act on a person's request are evaluated as well and shown in PST-1; they never change this band. - **Evidence kept:** Raw `robots.txt`, the parsed rule tree, a per-crawler allow/deny verdict for `/` and for three sampled content URLs, and which group each verdict came from. - **Bands:** anchored on *which* crawlers are blocked, not how many. The five major search crawlers are `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Googlebot` and `bingbot`; the other four are `Applebot`, `MistralAI-Index`, `meta-webindexer` and `Amzn-SearchBot`. - **4** — every one of the nine may fetch content paths. - **3** — one or two of the other four blocked; all five major crawlers allowed. - **2** — three or four of the other four blocked; all five major crawlers allowed. - **1** — exactly one major search crawler blocked. - **0** — `Disallow: /` for `*` with no named exception, or two or more major search crawlers blocked. - **Why the gate turns on the search crawlers:** the Index measures whether machines can find, read and *use* a site. A search or answer crawler blocked in `robots.txt` removes the site from that assistant, which is the failure the gate exists to catch. A training block is a rights choice, and this document declines to judge rights choices: PST-1 shows them and never scores them. A university that blocks `GPTBot` and welcomes `OAI-SearchBot` is reachable, and gating it would measure our approval rather than machine reach. - **Consequence:** blocking every training crawler costs no points and does not gate. Blocking a single major search crawler gates. - **Absent robots.txt:** a 404 scores 4. Nothing is disallowed, so nothing is blocked. A missing file is not a failure of this test. - **Note:** A deliberate, documented block is a business decision, not a defect. Record it, score it honestly, and say in the report that the client chose it. The number reflects machine reach, not our approval. - **Fix:** Remove inherited blocks. Keep only directives the client can explain. - **What changed in 12.0:** Names each company's search crawler (adds Claude-SearchBot, Googlebot, bingbot, Applebot, MistralAI-Index, meta-webindexer, Amzn-SearchBot). Training crawlers no longer cost points. Points: 5 in 11.5, 8 in 12.0. - **Replaces:** AIR-1.1 - **Review by:** 2027-04-18 ### RCH-2 — Does your website let those crawlers in, or does your security turn them away? - **Question:** Does your website let those crawlers in, or does your security turn them away? - **Points:** 12 - **Gate** - **Applicability:** always - **Verification:** auto - **Effort:** medium - **Fixed by:** Operations, Back-end developer - **Evidence:** AI product - **Measures:** We visit the site the way each crawler would and see whether your firewall or CDN refuses it, slows it down or shows a challenge page it cannot pass. - **Why this matters:** A firewall or bot rule can turn a crawler away even when robots.txt says yes, or put a challenge page in front of it that it cannot pass. We request pages the way each crawler does and record what comes back. [Cloudflare, new AI traffic options](https://blog.cloudflare.com/content-independence-day-ai-options/): Cloudflare sits in front of a large share of the web, and this is its own account of the crawler settings most sites actually enforce. - **Sources:** Cloudflare, 'Perplexity is using stealth, undeclared crawlers' (address to confirm), 2025-08-04 (A); [Cloudflare, new AI traffic options](https://blog.cloudflare.com/content-independence-day-ai-options/), 2026-07-01 (A); [RESONEO, LLM crawler test](https://think.resoneo.com/sentinel/geo-llm-crawler-report.html), 2026-01 (B) - **Detect:** Two halves, both read from the same probe. **The edge half:** request 10 sampled URLs with each search crawler's real User-Agent string (the nine named in RCH-1) and with the user-triggered fetchers whose operators publish a user-agent string (`ChatGPT-User`, `Claude-User`, `Perplexity-User`). Record status code, response size, and whether the body matches the baseline fetch. Flag 403, 429, 503 and bodies more than 40% smaller than baseline. Training crawlers are probed too, on three URLs, and shown in PST-1; a block of them is a rights choice and does not change this band. Cloudflare publishes no response header that names its Search, Agent and Training settings, so where responses come through Cloudflare the scan reports, per class, whether the crawlers were served or turned away, and says that this is read from the pattern and not from a setting. **The challenge half:** request browse, search, listing and detail pages with the scanner's own identity and look for Turnstile, reCAPTCHA, hCaptcha and JavaScript-challenge signatures, meta-refresh interstitials and challenge cookies in the raw response, on every URL that is not a form or a login. - **Evidence kept:** A crawler × URL matrix of status codes, sizes and challenge detection, and the matching signature for every challenge found. - **Bands:** the weakest half decides the band. - **4** — every crawler gets `200` with full content, and no read-only page carries a challenge. - **3** — one crawler rate-limited but eventually served. - **2** — one crawler blocked or challenged, or a challenge on a minority of read-only pages. - **1** — several crawlers blocked or challenged, or a challenge on most read-only pages. - **0** — most crawlers blocked. - **Note:** The wire-level `User-Agent` on these requests carries another company's product token. *Fetching*, above, states the three constraints that make that measurement rather than impersonation. - **Fix:** Audit Cloudflare bot settings, Pantheon AGCDN rules, Fastly VCL and origin WAF. Scope challenges to `POST` endpoints and authenticated routes. Verify by re-running this probe, not by reading configuration. - **What changed in 12.0:** Absorbs the challenge-page check. Reads Cloudflare's Search, Agent and Training settings where they can be seen. Points: 6 in 11.5, 12 in 12.0. - **Replaces:** AIR-1.2, AIR-1.3 - **Review by:** 2027-04-18 ### RCH-3 — Does each page say which web address is the real one? - **Question:** Does each page say which web address is the real one? - **Points:** 2 - **Applicability:** always - **Verification:** auto - **Effort:** small - **Fixed by:** Back-end developer, Search specialist - **Evidence:** Search index - **Measures:** We look for the canonical link on the page and check that it points to a working address that agrees with itself. - **Why this matters:** A canonical link names the true address of a page. Without one, the same content can read as several competing pages. [Google, canonicalization](https://developers.google.com/search/docs/crawling-indexing/canonicalization): Google explains how its index decides which address is real, and its AI answers draw on that index. - **Sources:** [Google, canonicalization](https://developers.google.com/search/docs/crawling-indexing/canonicalization), 2026-08-20 (A); [Google, AI features and your website](https://developers.google.com/search/docs/appearance/ai-features), 2025-12-10 (A) - **Detect:** For each sampled page, extract the canonical and resolve it. Flag: missing, multiple, cross-host, pointing at a redirect, pointing at a non-200, or pointing at the homepage from a deep page. - **Evidence kept:** Per page: the canonical found, its resolved status, and the flag raised. - **Fix:** Emit a self-referencing canonical by default and correct the exceptions. - **What changed in 12.0:** Detection and bands unchanged. Points: 2, unchanged from 11.5. - **Replaces:** AIR-1.7 - **Review by:** 2027-04-18 ### RCH-4 — Do your page settings let assistants quote you? - **Question:** Do your page settings let assistants quote you? - **Points:** 3 - **Applicability:** always - **Verification:** auto - **Effort:** small - **Fixed by:** Search specialist, Back-end developer - **Evidence:** AI product - **Measures:** We look for tags and headers that keep a page out of search or out of AI answers, such as noindex, nosnippet, noarchive and nocache, and whether they sit on pages you want cited. - **Why this matters:** Page tags and headers that stop a search engine from showing a snippet, or from indexing a page at all, also keep that page out of the AI answers built on the search index. [Google, AI features and your website](https://developers.google.com/search/docs/appearance/ai-features): Google documents that its snippet controls apply to its AI features, so a stray tag has a consequence there. - **Sources:** [Google, robots meta tag and X-Robots-Tag](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag), 2026-03-24 (A); [Google, AI features and your website](https://developers.google.com/search/docs/appearance/ai-features), 2025-12-10 (A); [Bing Webmaster Blog, usage controls (noarchive, nocache)](https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat), 2023-09-22 (A); [Apple, About Applebot](https://support.apple.com/en-us/119829), 2026-09-04 (A) - **Detect:** Parse `` and the bot-specific meta names (`googlebot`, `bingbot`, `applebot`) and the `X-Robots-Tag` header, including its user-agent-prefixed form, across the sample. Flag `noindex`, `nosnippet`, `max-snippet:0`, `noarchive` and `nocache`, and say which crawler each is addressed to. A directive on a page that is meant to be cited counts against it; pages that are deliberately private (login, account, cart, internal search results, confirmation pages) are left out of the sample for this test. - **Evidence kept:** Per page: every robots directive found, where it came from (tag or header), and the crawler it addresses. - **Bands:** Default coverage rubric, where a page passes only if no directive removes it from search or from AI answers. - A site-wide `noindex` or `nosnippet` is **band 0**, whatever the sample says. - **Note:** `noarchive` is counted because Bing documents it as live. Google removed its cache, so the Index no longer reasons from `noarchive` for Google. - **Fix:** Remove the directives from pages you want cited. They are almost always left over from a legacy SEO module or a staging template. - **What changed in 12.0:** Rewritten around Google, Bing and Apple, and now also catches noindex and a zero snippet length. noarchive is live for Bing. Points: 1 in 11.5, 3 in 12.0. - **Replaces:** AIR-1.6 - **Review by:** 2027-04-18 ## #2 · Read · 19 tests · 35 pts **Points 35** Reach asks whether an AI system can get in. Read asks whether it can find the pages and read the words once it has. The tests of finding pages (sitemaps, redirects, missing pages, filters, pagination, file rules) sit here with the tests of reading them (the text in the first response, Markdown copies, page structure), because both decide whether a page is usable once somebody has the address. ### RED-1 — Is your page's text in the page itself, without needing JavaScript? - **Question:** Is your page's text in the page itself, without needing JavaScript? - **Points:** 10 - **Applicability:** always - **Verification:** auto - **Effort:** large - **Fixed by:** Front-end developer, Back-end developer - **Evidence:** AI product - **Measures:** We compare the text in the first response from your server with the text a browser shows after scripts run. Most AI crawlers only ever see the first one. - **Why this matters:** If your text only appears after JavaScript runs, a tool that does not run it sees an empty page. This is the heaviest test in the Index. [Anthropic, web fetch tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool): Anthropic documents that its fetch tool does not run JavaScript, which is exactly the behavior this test measures. - **Sources:** [Anthropic, web fetch tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool), current (A); [Vercel and MERJ, 'The rise of the AI crawler'](https://vercel.com/blog/the-rise-of-the-ai-crawler), 2024-12-17 (B); [RESONEO, LLM crawler test](https://think.resoneo.com/sentinel/geo-llm-crawler-report.html), 2026-01 (B); [Google, JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics), 2026-03-04 (A) - **Detect:** For every sampled URL, extract main-content text from the raw fetch and from the rendered fetch. Compute `coverage = raw_chars ÷ rendered_chars` after normalizing whitespace and stripping nav, header and footer regions, and report its complement, the share of words missing from the first response, beside it. Average across the sample and report the per-template distribution. - **Evidence kept:** Per-URL character counts, coverage ratio, and the first 500 characters of each version for spot-checking. - **Bands:** mean coverage. - **4** — mean coverage ≥0.95. - **3** — 0.80–0.94. - **2** — 0.50–0.79. - **1** — 0.20–0.49. - **0** — below 0.20. - **Edge cases:** Coverage above 1.0 happens when JavaScript *removes* content that the raw response contained. Clamp the per-URL value to 1.0 for banding and keep the unclamped ratio in the evidence. A URL whose rendered text is empty is excluded from the mean and the exclusion is recorded: it is a fetch problem, not a coverage measurement. - **Note:** Report the worst-performing template by name. "Your program pages ship 8% of their content" lands harder than a mean. Google's and Apple's crawlers render JavaScript and several AI fetchers do not; date the claim, because it is changing. - **Fix:** Server-render or statically render. For decoupled Drupal this is the largest line item in most remediations. - **What changed in 12.0:** Measures the share of words missing from the first response instead of pass or fail. Points: 6 in 11.5, 10 in 12.0. - **Replaces:** AIR-2.1 - **Review by:** 2027-04-18 ### RED-2 — Is the text behind tabs and 'read more' buttons already on the page? - **Question:** Is the text behind tabs and 'read more' buttons already on the page? - **Points:** 5 - **Applicability:** conditional — the site uses tabs, accordions, load-more or infinite scroll - **Verification:** auto - **Effort:** medium - **Fixed by:** Front-end developer - **Evidence:** AI product - **Measures:** We check that text hidden behind tabs, accordions and 'read more' buttons is in the first response and is not removed from the page until someone clicks. - **Why this matters:** Tabs and accordions are fine, as long as their text is already in the page before anybody clicks. [Anthropic, web fetch tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool): The same tool does not run JavaScript, so anything that only appears after a click is out of its reach. - **Sources:** [Anthropic, web fetch tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool), current (A); [RESONEO, LLM crawler test](https://think.resoneo.com/sentinel/geo-llm-crawler-report.html), 2026-01 (B); [Google, control your snippets](https://developers.google.com/search/docs/appearance/snippet), 2026-04-20 (A) - **Detect:** Identify disclosure widgets by ARIA roles (`tab`, `tabpanel`, `region` with `aria-expanded`), `
` and common class patterns. For each, check that the panel body is populated in the raw HTML **and** that the rendered page still holds it: text that a script removes from the page until someone clicks fails. Detect infinite scroll by watching for XHR content growth on scroll in the rendered fetch, and check for a paginated fallback. - **Evidence kept:** Per widget: the pattern found, whether the panel body is in the raw HTML and in the rendered page. - **Bands:** 4 — every panel populated in raw HTML and kept in the page; infinite scroll has crawlable pagination. 3 — one pattern incomplete. 2 — panels populated on some templates only. 1 — most content loads on interaction. 0 — all disclosure content is fetched on demand. - **Note:** Text present in the HTML and hidden with CSS passes. One independent test reports that some tools may still drop CSS-hidden text; that result is on the evidence register's open list, and the rule tightens if it is confirmed. - **Fix:** Render all panels and hide with CSS, never by removing the text. Give infinite scroll a paginated fallback. - **What changed in 12.0:** Asks for visible on load, not only present. Points: 3 in 11.5, 5 in 12.0. - **Replaces:** AIR-2.2 - **Review by:** 2027-04-18 ### RED-3 — Can a crawler trust the 'last changed' dates in your sitemap? - **Question:** Can a crawler trust the 'last changed' dates in your sitemap? - **Points:** 2 - **Applicability:** always - **Verification:** auto-partial - **Effort:** medium - **Fixed by:** Back-end developer, Search specialist - **Evidence:** AI product - **Measures:** We compare the dates in your sitemap with whether the pages actually changed. - **Why this matters:** A last-changed date should mean the content changed, not that you deployed. Once it is noise, crawlers stop trusting it. [Bing Webmaster Blog, sitemaps in AI-powered search](https://blogs.bing.com/webmaster/July-2025/Keeping-Content-Discoverable-with-Sitemaps-in-AI-Powered-Search): Bing ties these dates to how quickly updates reach its search and AI answers. - **Sources:** [Bing Webmaster Blog, sitemaps in AI-powered search](https://blogs.bing.com/webmaster/July-2025/Keeping-Content-Discoverable-with-Sitemaps-in-AI-Powered-Search), 2025-07-31 (A); [Google, build and submit a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap), 2026-07-08 (A) - **Detect:** Parse all `lastmod` values. Flag clustering: more than 60% of URLs sharing a single date, or timestamps identical to the minute across content types. Cross-check against on-page `dateModified` where present. Re-run across two audits to confirm. - **Evidence kept:** The `lastmod` distribution, the clustering found and the on-page dates compared. - **Bands:** 4 — values are distributed and match on-page dates. 3 — mostly accurate with some clustering. 2 — heavy clustering. 1 — all identical. 0 — absent. - **First run:** cap at 3 without a baseline run. See *Tests that need a previous run*. - **Fix:** Wire `lastmod` to the node's changed timestamp, not to cron. - **What changed in 12.0:** Bing ties sitemap dates to how fast updates reach AI answers, which is why the test stays in the score. Points: 2, unchanged from 11.5. - **Replaces:** AIR-4.8 - **Review by:** 2027-04-18 ### RED-4 — Do your filters and calendars keep a crawler from getting lost? - **Question:** Do your filters and calendars keep a crawler from getting lost? - **Points:** 1 - **Applicability:** conditional — the site has search, filtering or a calendar - **Verification:** auto-partial - **Effort:** medium - **Fixed by:** Search specialist, Back-end developer - **Evidence:** Search index - **Measures:** We look for endless combinations of links, such as filter and date pages, that waste a crawler's visit. - **Why this matters:** Filters and calendars can generate millions of addresses. A crawler spends its visit there instead of on the pages you care about. [Google, canonicalization](https://developers.google.com/search/docs/crawling-indexing/canonicalization): Google explains how it groups duplicate and parameterized addresses, which is the waste filters and calendars create. - **Sources:** [Google, canonicalization](https://developers.google.com/search/docs/crawling-indexing/canonicalization), 2026-08-20 (A) - **Detect:** Crawl listing pages two levels deep with parameters followed. Measure URL growth rate and count distinct parameter combinations. Check whether those URLs are disallowed, `noindex`, or canonicalized to the unfiltered listing. - **Evidence kept:** Growth per level, the parameter combinations counted and the control found for each. - **Bands:** 4 — parameter space bounded by robots rules, `noindex` or canonicals. 3 — mostly controlled with gaps. 2 — partially controlled. 1 — minimal control. 0 — unbounded expansion observed. - **Fix:** Combine disallow patterns, `noindex` on parameterized views, and canonicals back to the base listing. - **What changed in 12.0:** Detection and bands unchanged. Points: 2 in 11.5, 1 in 12.0. - **Replaces:** AIR-4.7 - **Review by:** 2027-04-18 ### RED-5 — Do your PDFs and images have their own rules for search and quoting? - **Question:** Do your PDFs and images have their own rules for search and quoting? - **Points:** 1 - **Applicability:** conditional — the site serves PDFs, documents, media or JSON endpoints - **Verification:** auto - **Effort:** small - **Fixed by:** Operations - **Evidence:** Search index - **Measures:** We check the X-Robots-Tag header on files that are not web pages. - **Why this matters:** PDFs, images and JSON cannot carry a meta robots tag, so the instruction has to travel in the HTTP header instead. [Google, robots meta tag and X-Robots-Tag](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag): Google documents the header for files that cannot hold a tag, including the snippet controls that apply to its AI features. - **Sources:** [Google, robots meta tag and X-Robots-Tag](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag), 2026-03-24 (A) - **Detect:** HEAD every sampled PDF, document, media file and API endpoint. Record `X-Robots-Tag`. Flag `noindex` on substantive content and absent directives on endpoints that should be excluded. - **Evidence kept:** Per resource: the type, the header found and the flag raised. - **Bands:** 4 — every non-HTML type carries a deliberate, correct value. 2 — mixed or partial. 0 — absent everywhere, or substantive PDFs marked `noindex`. - **Fix:** Set headers by path pattern at the CDN or web server. - **What changed in 12.0:** Detection and bands unchanged. Points: 1, unchanged from 11.5. - **Replaces:** AIR-4.4 - **Review by:** 2027-04-18 ### RED-6 — Do missing pages say they are missing? - **Question:** Do missing pages say they are missing? - **Points:** 1 - **Applicability:** always - **Verification:** auto - **Effort:** small - **Fixed by:** Back-end developer, Operations - **Evidence:** Search index - **Measures:** We ask for pages that do not exist and check the answer is a real 404 or 410, not a normal-looking page. - **Why this matters:** A missing page that answers 200 OK looks like a real page, and a crawler spends its visit on it. [Google, HTTP status codes](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes): Google lists how its crawler treats each status code, including pages that should have been errors. - **Sources:** [Google, HTTP status codes](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes), 2026-02-04 (A); [Vercel and MERJ, 'The rise of the AI crawler'](https://vercel.com/blog/the-rise-of-the-ai-crawler), 2024-12-17 (B) - **Detect:** Request 10 deliberately invalid URLs under real path prefixes. Flag `200` responses. Separately, flag sampled pages whose body matches a not-found template or falls below a minimum content threshold while returning `200`. - **Evidence kept:** Probe URLs with status codes and body fingerprints. - **Note:** One study found AI crawlers spent about a third of their visits on missing pages. The claim here is limited to wasted visits. - **Fix:** Return real status codes. Use `410` for content that is permanently gone. - **What changed in 12.0:** Citation updated; the claim is limited to wasted visits. Points: 1, unchanged from 11.5. - **Replaces:** AIR-4.5 - **Review by:** 2027-04-18 ### RED-7 — Do your redirects go straight to the right page? - **Question:** Do your redirects go straight to the right page? - **Points:** 1 - **Applicability:** always - **Verification:** auto - **Effort:** small - **Fixed by:** Back-end developer, Search specialist - **Evidence:** Search index - **Measures:** We follow your redirects and count the hops. - **Why this matters:** Every extra redirect is another chance for a crawler to give up. One hop, straight to the destination. [Google, HTTP status codes](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes): Google states how many redirect hops its crawler follows, which is the limit this test works inside. - **Sources:** [Google, HTTP status codes](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes), 2026-02-04 (A) - **Detect:** Follow redirects without auto-resolution and count hops per sampled URL and per internal link target. - **Evidence kept:** Per URL: the chain, its length and whether it loops. - **Bands:** 4 — no chain exceeds one hop. 3 — a few two-hop chains. 2 — chains of three or more. 1 — widespread chains, meaning both more than 10% of the sample and at least three URLs. 0 — loops present. - **Note:** Google follows up to ten hops. That is a limit, not a recommendation, and the test asks for one hop because every hop is another request. - **Fix:** Flatten the redirect table so every source points at its final destination. - **What changed in 12.0:** Citation updated to Google's ten-hop limit. Points: 1, unchanged from 11.5. - **Replaces:** AIR-4.6 - **Review by:** 2027-04-18 ### RED-8 — Do you have sitemaps for your images and videos? - **Question:** Do you have sitemaps for your images and videos? - **Points:** 1 - **Applicability:** always - **Verification:** auto - **Effort:** small - **Fixed by:** Back-end developer, Search specialist - **Evidence:** Search index - **Measures:** We look for image and video sitemaps. - **Why this matters:** Image and video sitemaps tell a crawler about media it would not find in the page text. [Google, build and submit a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap): Google's own instructions for the sitemap extensions its crawler reads. - **Sources:** [Google, build and submit a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap), 2026-07-08 (A) - **Detect:** Fetch the sitemaps named in `robots.txt` and at `/sitemap.xml`. Look for image and video sitemap entries, or child sitemaps of those types, where the sampled pages carry substantial media. Validate against the sitemap schema and the 50,000-URL and 50 MB limits. - **Evidence kept:** The sitemaps found, the media types they cover and any limit exceeded. - **Bands:** 4 — the sampled pages carry no substantial media, or image and video sitemaps are present and valid. 2 — present but invalid or partial. 0 — substantial media and no media sitemap. - **Note:** The claim that splitting a sitemap by content type helps was dropped: it is a Search Console convenience, and no source ties it to AI answers. - **Fix:** Generate image and video sitemaps from the media library and submit them to both search consoles. - **What changed in 12.0:** The claim that splitting a sitemap by content type helps is dropped. Points: 1, unchanged from 11.5. - **Replaces:** AIR-4.9 - **Review by:** 2027-04-18 ### RED-9 — Can a crawler reach page two, three and beyond without clicking a button? - **Question:** Can a crawler reach page two, three and beyond without clicking a button? - **Points:** 1 - **Applicability:** conditional — the site has paginated listings or multi-page articles - **Verification:** auto - **Effort:** small - **Fixed by:** Search specialist, Back-end developer - **Evidence:** Search index - **Measures:** We check that lists of pages link to the next page with ordinary links, not only a 'load more' button. - **Why this matters:** Page two and everything after it has to be reachable with an ordinary link, because a crawler does not click a button. [Google, JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics): Google explains what its crawler does and does not do with JavaScript, including content that loads on a click. - **Sources:** [Google, JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics), 2026-03-04 (A); [Vercel and MERJ, 'The rise of the AI crawler'](https://vercel.com/blog/the-rise-of-the-ai-crawler), 2024-12-17 (B) - **Detect:** Identify paginated series by URL pattern. Check that each page links to the next with an ordinary link in the raw HTML and not only through a 'load more' button, that a view-all page carries the canonical where there is one, and that deep pages are reachable by crawl. `rel=next` and `rel=prev` are reported and not required. - **Evidence kept:** Per series: the link found, whether it is in the raw HTML, and the depth reached. - **Fix:** Prefer a view-all canonical where page weight allows, and give every list ordinary next-page links. - **What changed in 12.0:** Detection and bands unchanged. Points: 1, unchanged from 11.5. - **Replaces:** AIR-4.10 - **Review by:** 2027-04-18 ### RED-10 — Do you offer a plain-text (Markdown) copy of your pages? - **Question:** Do you offer a plain-text (Markdown) copy of your pages? - **Points:** 2 - **Applicability:** always - **Verification:** auto - **Effort:** medium - **Fixed by:** Back-end developer - **Evidence:** AI product - **Measures:** We look for a Markdown copy of the page, at the address the page advertises or at the page's address with .md added. - **Why this matters:** A plain-text Markdown copy of a page has none of the navigation and markup a tool has to wade through to reach your words. [Cloudflare, Markdown for Agents](https://developers.cloudflare.com/changelog/post/2026-02-12-markdown-for-agents/): Cloudflare documents serving Markdown to agents that ask for it, from the network most sites sit behind. - **Sources:** [Cloudflare, Markdown for Agents](https://developers.cloudflare.com/changelog/post/2026-02-12-markdown-for-agents/), 2026-02-12 (A); [Dries Buytaert, a month of Cloudflare logs](https://dri.es/markdown-llms-txt-and-ai-crawlers), 2026-03-05 (B) - **Detect:** For each URL in the representation sub-sample, find the Markdown copy the page advertises (a `` or a `Link` header) and request it; where the page advertises none, request `{path}.md`. Require a `200` with `text/markdown` or `text/plain`, content that materially matches the page body, and no navigation chrome. - **Evidence kept:** Per URL: the address tried, how it was found, status, type and a body comparison. - **Fix:** Generate from the same source as the HTML. Strip nav, promos and cookie banners. - **What changed in 12.0:** Follows the address the page advertises, not only the page address plus .md. Points: 2, unchanged from 11.5. - **Replaces:** AIR-7.1 - **Review by:** 2027-04-18 ### RED-11 — Does each page tell machines where its Markdown copy is? - **Question:** Does each page tell machines where its Markdown copy is? - **Points:** 1 - **Applicability:** conditional — requires RED-10 ≥ 2 - **Verification:** auto - **Effort:** small - **Fixed by:** Front-end developer - **Evidence:** AI product - **Measures:** We look for the Markdown link in the page's head and in the response headers. - **Why this matters:** Publishing the Markdown copy is not enough: the page has to say where it is, or nothing will find it. [llms.txt specification v2 (Link relations)](https://llmstxt.org/): The specification defines how a page points to its Markdown copy, in the page and in a response header. - **Sources:** [llms.txt specification v2 (Link relations)](https://llmstxt.org/), 2026-08-10 (A); [RFC 8288, Web Linking](https://www.rfc-editor.org/rfc/rfc8288), 2017 (A); [Dries Buytaert, a month of Cloudflare logs](https://dri.es/markdown-llms-txt-and-ai-crawlers), 2026-03-05 (B) - **Detect:** Parse `` from the head and the `Link` response header (RFC 8288) for the same relation. Either form counts. Resolve the address and confirm it is the copy RED-10 found. - **Evidence kept:** Per URL: the head link and the header link found, and where each pointed. - **Fix:** Add the link to the head template or send it as a response header. A file nobody can find is not much use. - **What changed in 12.0:** Reads the response headers as well as the link in the page. Points: 2 in 11.5, 1 in 12.0. - **Replaces:** AIR-7.2 - **Review by:** 2027-04-18 ### RED-12 — If a machine asks for Markdown, does your site give it? - **Question:** If a machine asks for Markdown, does your site give it? - **Points:** 2 - **Applicability:** always - **Verification:** auto - **Effort:** medium - **Fixed by:** Back-end developer, Operations - **Evidence:** AI product (coding agents) - **Measures:** We send a request that asks for Markdown, check the answer, and check that caches keep the web and Markdown versions apart. - **Why this matters:** A machine can ask the page's own address for Markdown and get it, with caches keeping the web and Markdown versions apart. [Cloudflare, Markdown for Agents (docs)](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/): Cloudflare documents the request and its cache behavior, and reports which coding agents send it. - **Sources:** [Cloudflare, Markdown for Agents](https://developers.cloudflare.com/changelog/post/2026-02-12-markdown-for-agents/), 2026-02-12 (A); [Cloudflare, Markdown for Agents (docs)](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/), 2026-10-09 (A); [Cloudflare, cache and Vary](https://developers.cloudflare.com/cache/concepts/vary/), 2026-08-14 (A) - **Detect:** Two halves. **Negotiation:** request sampled URLs with `Accept: text/markdown` and confirm the response type and that `Vary: Accept` is set. **Caching:** request through the CDN twice, once for each representation, and confirm the two variants are cached separately and that a browser is never served Markdown. - **Evidence kept:** Per URL: response type, `Vary` header, cache-status headers and the variant returned to each request. - **Bands:** the weakest half decides the band. - **4** — negotiation works, `Vary: Accept` is correct and the variants are cached separately. - **2** — negotiation works but `Vary` is missing or the variants share a cache entry, which risks serving Markdown to browsers. - **0** — not supported. - **Note:** The evidence is for coding agents only: Cloudflare reports that they send the request, and one study of crawler logs saw no AI crawler do so. Getting `Vary` wrong is worse than not doing this at all. Verify at the edge. - **Fix:** Serve Markdown on request from the canonical address, set `Vary: Accept`, and set TTLs and invalidation to the same cache tags as the source content. - **What changed in 12.0:** Absorbs the caching test. Points: 4 in 11.5, 2 in 12.0. - **Replaces:** AIR-7.3, AIR-7.4 - **Review by:** 2027-04-18 ### RED-13 — Is the page built from real page parts, like header, main and navigation? - **Question:** Is the page built from real page parts, like header, main and navigation? - **Points:** 1 - **Applicability:** always - **Verification:** auto - **Effort:** medium - **Fixed by:** Front-end developer - **Evidence:** Practice - **Measures:** We check for real page-structure elements instead of one pile of unlabeled boxes. - **Why this matters:** Real page parts such as header, main and navigation, instead of one pile of unlabeled boxes. It is good practice for people, and research suggests structure helps a model read a page. [HtmlRAG (research paper)](https://arxiv.org/abs/2411.02959): The paper tests whether keeping HTML structure helps a model answer from a page, and it is the only evidence we found for this test. - **Sources:** [HtmlRAG (research paper)](https://arxiv.org/abs/2411.02959), 2024-11 (B); [Google, AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), 2026-07-10 (A) - **Detect:** Per page, require exactly one `
`, at least one `
` or `
` wrapping the primary content, and a `