What is AI web readiness?

AI web readiness measures how well a website can be discovered, parsed, and cited by AI agents and large language models. It covers signals like /llms.txt, structured data, robots.txt AI crawler permissions, server-side rendering, and OpenGraph metadata.

What is /llms.txt and why does it matter?

/llms.txt is a plain text file placed at the root of a website that tells AI agents what the site is about, what it offers, and which pages to prioritize. Without it, AI agents have to guess your site's structure and purpose. It is the most impactful single change you can make for AI visibility.

How is the AI readiness score calculated?

The score is calculated across 20 weighted checks, each representing a signal that AI agents use when reading, classifying, or citing a website. Each check is a pass, partial, or fail, and the points add up to a score out of 100. The checks cover indexability (10 pts), /llms.txt (10 pts), structured data (9 pts), robots.txt AI permissions (8 pts), and further signals totalling 100 core points, plus four optional bonus checks.

Is AI readiness the same as SEO?

They overlap but are distinct. Traditional SEO optimizes for search engine crawlers and ranking algorithms. AI readiness optimizes for large language models and AI agents that read, summarize, and cite content directly. Many SEO best practices (structured data, fast load times, clear content) also help AI readiness, but AI-specific signals like /llms.txt have no SEO equivalent.

Which AI crawlers does the tool check for?

The tool checks for explicit permissions in robots.txt for GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, anthropic-ai, cohere-ai, and meta-externalagent, among others.

Is this tool free to use?

Yes, AI Web Readiness is completely free to use. No account or sign-up is required. Simply enter a URL and get a full report in seconds.

censoredjournalismcom.netlify.app - 49/100 Fair

expand

Grade

Fair

Agents can find you, but key signals are missing. Worth a focused pass.

Score

+0 / 7 bonus

Passed

Partial

Failed

Not sure where to start?

Get a free 15-minute session with an AI visibility expert. We'll walk through your results and prioritise the fixes that move the needle fastest.

Book a free call →

No obligation · 15 min · Free

Check breakdown6 passed · 2 partial · 12 failed

AI agent manifest

No /llms.txt found (status 404).

0 / 10

⌄

llms.txt quality

There is no usable /llms.txt to evaluate. Add one first.

0 / 6

⌄

Full-content AI feed

No /llms-full.txt found (status 404).

0 / 3

⌄

✓

Indexability

The homepage can be indexed: no noindex directive, and robots.txt does not block search crawlers.

10 / 10

⌄

Structured data

No JSON-LD structured data found on the homepage.

0 / 9

⌄

Attribution schema

No Organization or Person entity, and no author or publisher property, found in JSON-LD structured data.

0 / 5

⌄

✓

Readable content ratio

1711 chars of text, 11.9% text-to-HTML ratio.

7 / 7

⌄

✓

Crawler-visible content

1711 chars server-rendered, 1 H1, 4 H2, 6 substantial paragraph(s).

8 / 8

⌄

AI crawler permissions

No /robots.txt found (status 404). Crawlers assume they may crawl, but nothing is declared and there is no Sitemap line. Publish a /robots.txt to earn full marks.

5 / 8

⌄

Page index

No /sitemap.xml (status 404).

0 / 6

⌄

Social and citation metadata

No OpenGraph tags found.

0 / 4

⌄

Page identity signals

title: 71 chars, description: 91 chars. Note: the title is long enough to be truncated. Titles of 40-70 and descriptions of 50-160 characters score full marks.

6.5 / 7

⌄

Canonical authority

No <link rel="canonical"> tag found.

0 / 4

⌄

✓

Redirect and host health

Good: http redirects to https, no redirect needed.

4 / 4

⌄

✓

Accessibility and speed

HTTPS confirmed, homepage downloaded in 60ms.

3 / 3

⌄

✓

Semantic structure

One H1, an ordered heading hierarchy, a <main> landmark, a lang attribute, alt text on images, and a viewport tag.

6 / 6

⌄

Q&A schemabonus

No FAQPage or HowTo schema found. Adding structured Q&A can significantly increase AI citations.

0 / 2

⌄

Content feedbonus

No RSS or Atom feed detected. A content feed signals an actively updated site that agents should re-index.

0 / 1

⌄

Agent discovery endpointsbonus

No agent discovery documents found (MCP server card, A2A agent card, API catalog, agents.json, or OpenAPI description).

0 / 2

⌄

Content freshnessbonus

No machine-readable publish or modified dates found (JSON-LD, meta tags, <time> elements, or sitemap <lastmod>).

0 / 2

⌄

Fix this with AI

A complete brief covering all 14 outstanding issues, with the reason each one matters and the exact fix. Copy it into Claude, ChatGPT, Cursor, or any coding agent working on censoredjournalismcom.netlify.app.

# AI readiness fixes for censoredjournalismcom.netlify.app

Audit by AI Web Readiness (https://aiwebreadiness.com) on 2026-10-15. Full report: https://aiwebreadiness.com/report/sXSBfvJF7k
Current score: 49/100 (Fair). 6 of 20 checks at full marks.

## Your task

You are working on the website https://censoredjournalismcom.netlify.app. An automated audit measured how well this site can be discovered, parsed, and cited by AI agents and LLM-powered search engines. The issues it found are listed below, ordered by how many points each is worth.

Work through them in order, highest value first. For each issue, make the change in this site's codebase or CMS using the stack and conventions already in use. Where a fix needs copy, facts, or URLs you cannot verify from the existing site, ask me rather than inventing them.

There are 50.5 points on the table across these 10 issues. Fixing all of them reaches 100/100. If you only have time for the top 3, that alone is worth 25 points and takes the score to 74/100.

---

## Issues to fix (10)

### 1. AI agent manifest (10 points available)

**What the scan found:** No /llms.txt found (status 404).

**Why it matters:** An llms.txt file is a short, plain-text briefing written for AI agents. It states what your site is, what you offer and which pages matter most, so an agent can understand you in seconds instead of crawling dozens of pages and guessing. Without it, an assistant has to infer your purpose from whatever it happens to find, and it may describe you inaccurately or skip you.

**How to fix it:** Create a plain text file at /llms.txt in your site root. It should start with a # Title line, a short > description, and a list of your key pages as markdown links. Use the draft generated below as your starting point. It is ready to deploy as-is.

### 2. Structured data (9 points available)

**What the scan found:** No JSON-LD structured data found on the homepage.

**Why it matters:** JSON-LD states plainly, in a format machines read without guessing, who you are, what you offer and how your pages relate to each other. Search engines and AI systems use it to build their understanding of an entity and to decide what to show or quote. Without it, they must infer meaning from layout and wording, and the inference is often wrong or shallow. Richer, more specific types give agents more facts they can use with confidence.

**How to fix it:** Add a <script type="application/ld+json"> block to your homepage <head>. At minimum include @context, @type (e.g. Organization, WebSite, or Article), name, url, and description. For higher scores add richer types like BreadcrumbList, Article, or Product on the relevant pages. Google's Structured Data Markup Helper is a good starting point.

### 3. llms.txt quality (6 points available)

**What the scan found:** There is no usable /llms.txt to evaluate. Add one first.

**Why it matters:** Agents rely on that structure to decide where to go next. A file that is just a paragraph of prose, or one full of broken or off-site links, gives them little to act on and can send them to dead ends. A well-formed file works like a trustworthy table of contents: it tells an assistant which pages answer which questions, which makes it more likely your best pages are the ones that get read and cited.

**How to fix it:** Follow the llmstxt.org format. Start with a single "# Site name" line, add a "> one sentence summary" blockquote, then group at least three markdown links under "## Section" headings, written as "- [Page name](https://yoursite.com/page): what it covers". Keep links on your own domain and make sure each one loads.

### 4. Page index (6 points available)

**What the scan found:** No /sitemap.xml (status 404).

**Why it matters:** A sitemap is the list of pages you want found. It lets crawlers discover content that internal links might not surface and tells them which pages have changed recently, so updates are picked up faster. Without one, discovery depends on crawlers following links, and pages that are deep in your site, new or poorly linked can be missed for a long time.

**How to fix it:** Create a valid XML sitemap listing your important pages with <loc> and <lastmod> entries, serve it at /sitemap.xml, and add a line 'Sitemap: https://yourdomain.com/sitemap.xml' to your robots.txt. Most CMS platforms generate this automatically.

### 5. Attribution schema (5 points available)

**What the scan found:** No Organization or Person entity, and no author or publisher property, found in JSON-LD structured data.

**Why it matters:** When an AI assistant quotes or recommends a source, it needs to know who stands behind the content. Clear attribution data lets it credit you by name, connect your site to your profiles and reputation, and trust that the content comes from a real, identifiable author or organization. Anonymous content is easier to ignore, and harder to cite with confidence.

**How to fix it:** Add an Organization (or Person) entity to your JSON-LD with a name, a url, a logo, and sameAs links to your official profiles (LinkedIn, Wikipedia, social accounts). On articles and content pages, add author and publisher properties that reference those entities.

### 6. Social and citation metadata (4 points available)

**What the scan found:** No OpenGraph tags found.

**Why it matters:** OpenGraph tags are the headline, summary and picture that others see when your page is shared, previewed or cited. AI assistants and link-preview systems use them to describe your page quickly and consistently. When they are missing, the tool has to improvise from the page body, and the result is often an awkward title, a wrong summary or no image at all.

**How to fix it:** Add OpenGraph meta tags to your page <head>: og:title (your page title), og:description (a 1-2 sentence summary), og:image (a representative image URL), and og:site_name (your brand name). Each tag present adds to your score. These same tags are used by social platforms when your content is shared.

### 7. Canonical authority (4 points available)

**What the scan found:** No <link rel="canonical"> tag found.

**Why it matters:** The same page is often reachable at several addresses, such as with and without www, with tracking parameters, or through old links. The canonical tag tells search engines and agents which address is the real one so that your authority is gathered in one place. A missing tag splits that authority, and a wrong one can tell search engines to drop your pages in favour of an address that does not exist.

**How to fix it:** Add <link rel="canonical" href="https://yourdomain.com/page-url"> to the <head> of every page. Use an absolute URL on your own domain that returns a 200 and matches the page itself. Never point it at a staging, old, or unrelated domain.

### 8. Full-content AI feed (3 points available)

**What the scan found:** No /llms-full.txt found (status 404).

**Why it matters:** Where llms.txt is a map, llms-full.txt is the whole territory: the full text of your key pages in one place. An agent with limited time or a limited context window can read your content in a single request, with no page-by-page crawling and no risk of missing something behind navigation or scripts. It is a small file to publish, and it lets AI tools represent your content accurately and completely.

**How to fix it:** Create /llms-full.txt at your site root containing the full text of your most important pages, concatenated in order. Each page should be separated by a clear heading. Many static site generators and CMS platforms have plugins that generate this automatically.

### 9. AI crawler permissions (3 points available, currently scoring 5 of 8)

**What the scan found:** No /robots.txt found (status 404). Crawlers assume they may crawl, but nothing is declared and there is no Sitemap line. Publish a /robots.txt to earn full marks.

**Why it matters:** Robots.txt is the first thing a well-behaved crawler reads, and a block there means the agent never sees your content at all. Some sites block AI agents on purpose, which is a legitimate choice, but many do it by accident through an overly broad rule or a security plugin. This check shows the real consequence: every agent that is shut out cannot read, learn from or cite your site.

**How to fix it:** Do not Disallow: / for the AI crawlers you want reading your site, and do not block them with a blanket User-agent: * rule. Optionally name them explicitly with Allow: / rules (GPTBot, ClaudeBot, PerplexityBot, Google-Extended). Blocking an agent is a legitimate choice, but it lowers this score. Publish a /robots.txt even if it only allows everything and points at your sitemap.

### 10. Page identity signals (0.5 points available, currently scoring 6.5 of 7)

**What the scan found:** title: 71 chars, description: 91 chars. Note: the title is long enough to be truncated. Titles of 40-70 and descriptions of 50-160 characters score full marks.

**Why it matters:** The title and description are the first things a search engine or an AI agent reads to decide what a page is about and whether it is relevant to a question. Missing or vague ones make the page harder to classify, and overlong ones get truncated in search results and citations. A clear, specific title and description improve how often your page is chosen and how accurately it is described.

**How to fix it:** Write a descriptive <title> tag of 40-70 characters that clearly names the page and your brand. Add a <meta name="description"> of 50-160 characters that summarises what the page offers. Longer values get truncated. Both should be unique per page, specific rather than generic, and written for humans.

---

## Optional bonus checks

These are scored separately from the 100-point core and do not change the headline score. They only apply if they fit the site. Skip any that do not make sense for what this site is.

- **Q&A schema** (up to 2 more points): No FAQPage or HowTo schema found. Adding structured Q&A can significantly increase AI citations. How: On any page that contains questions and answers, add a FAQPage JSON-LD block with Question and Answer entities. Each Question needs a name (the question text) and an acceptedAnswer with a text property. For step-by-step content, use HowTo schema instead. Google's Rich Results Test can validate your implementation.
- **Agent discovery endpoints** (up to 2 more points): No agent discovery documents found (MCP server card, A2A agent card, API catalog, agents.json, or OpenAPI description). How: Publish at least one machine-readable discovery document that applies to you, for example /.well-known/mcp.json for an MCP server, /.well-known/agent-card.json for an A2A agent, /.well-known/api-catalog for your APIs, or /openapi.json for a REST API. Two or more earn full marks.
- **Content freshness** (up to 2 more points): No machine-readable publish or modified dates found (JSON-LD, meta tags, <time> elements, or sitemap <lastmod>). How: Expose dates in machine-readable form: datePublished and dateModified in your JSON-LD, article:modified_time meta tags, <time datetime> elements, and <lastmod> in your sitemap. Keep them truthful and update them when content changes.
- **Content feed** (up to 1 more points): No RSS or Atom feed detected. A content feed signals an actively updated site that agents should re-index. How: Create an RSS or Atom feed at /feed.xml or /rss.xml listing your most recent content with title, link, description, and pubDate per item. Add a <link rel="alternate" type="application/rss+xml" href="/feed.xml"> tag to your <head> so agents can discover it automatically. A feed that exists but is not advertised earns partial credit.

---

## Ready-to-deploy /llms.txt

This draft was generated from the site's own metadata and links. Review the wording, correct anything inaccurate, then save it as a plain text file at the site root so it resolves at https://censoredjournalismcom.netlify.app/llms.txt

```
# Censored Journalism: sourced explainers on Pakistan and the Middle East

> Sourced explainers on Pakistan and the Middle East. Every claim has a source you can check.

## Key Pages
- [Home](https://censoredjournalismcom.netlify.app/)

## About
This file follows the /llms.txt convention, which provides AI agents and language models with a structured map of this site. Learn more at https://llmstxt.org.
```

---

## Already passing, do not regress

These signals are already at full marks. Preserve them while making the changes above:

- **Indexability** (10 pts): The homepage can be indexed: no noindex directive, and robots.txt does not block search crawlers.
- **Readable content ratio** (7 pts): 1711 chars of text, 11.9% text-to-HTML ratio.
- **Crawler-visible content** (8 pts): 1711 chars server-rendered, 1 H1, 4 H2, 6 substantial paragraph(s).
- **Redirect and host health** (4 pts): Good: http redirects to https, no redirect needed.
- **Accessibility and speed** (3 pts): HTTPS confirmed, homepage downloaded in 60ms.
- **Semantic structure** (6 pts): One H1, an ordered heading hierarchy, a <main> landmark, a lang attribute, alt text on images, and a viewport tag.

---

## When you are done

Deploy the changes, then rescan censoredjournalismcom.netlify.app at https://aiwebreadiness.com to confirm the score moved. The report tracks the delta between scans automatically.

Click the box to select all · 13,406 characters

/llms.txt draft

Ready-to-deploy file. Drop it in your site root so AI agents can find a structured map of what you offer.

⌄