Website Audit: find what stops AI and search engines reading your site

An audit needs a brand name and a website address; a short description is optional, and a URL without https:// gets it added. If you haven't set up brand context yet, Intura drafts a knowledge profile and a target audience from your brand, description and site before any page is scored. The optional AI analysis uses them later to judge whether a page is on brand.

The crawl starts by reading robots.txt and finding your sitemaps, both from the Sitemap: lines in robots.txt and from the usual locations such as /sitemap.xml and /sitemap_index.xml. Sitemap index files, gzipped sitemaps and plain-text sitemaps are all read.

It then crawls breadth-first from the homepage, follows internal links up to five levels deep and obeys your robots.txt the way Googlebot would. Only pages that return HTTP 200, serve HTML and aren't blocked by robots.txt become audited pages.

By default, blog posts and product pages are capped at 100 each, so a large catalogue doesn't crowd out the rest. Your homepage, pricing, about, contact, FAQ, service and feature pages are always kept.

Some problems only exist at the level of the whole site. These checks run once per audit:

• robots.txt: missing, server errors, redirects, blocking the whole site, important paths or CSS and JS files, syntax errors, no sitemap listed.
• AI crawler access: whether robots.txt blocks AI agents, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and Meta's crawler.
• llms.txt: whether /llms.txt exists. A missing file lowers the site score; /llms-full.txt is optional and only raises a warning.
• Sitemaps: missing, unreadable, invalid or oversized; listing non-indexable or broken pages; missing pages the crawl found; cross-domain URLs; bad lastmod dates.
• HTTPS and server: no HTTPS, HTTP not redirected, an invalid or expiring certificate, www and non-www both live, no HSTS, soft 404s, no compression, no HTTP/2, a slow server, no favicon.

Every audited page goes through the full rule set: 126 rules in 12 categories. Each rule has a severity (critical, high, medium, low or info), a short explanation of why it matters, and the fix. The main groups:

• Status and redirects: 5xx, 404 and 410 errors, 401 and 403, redirect chains and loops, temporary redirects, meta refresh, slow responses, very deep URLs, orphan pages.
• Indexability: noindex, nofollow, conflicting robots directives, noindex pages that sit in the sitemap or are linked internally.
• Canonicals and duplicates: missing, multiple, relative or cross-domain canonicals, canonicals pointing at non-indexable pages, duplicate content, titles, descriptions and H1s.
• On-page content: missing, duplicated, too long or too short titles and meta descriptions; missing, multiple or empty H1; skipped heading levels; thin content under 200 words; a low text-to-HTML ratio; missing language, doctype or charset; no Open Graph or Twitter card.
• Structured data: invalid JSON-LD, no structured data, missing required properties, no BreadcrumbList, no Organization or WebSite schema on the homepage.
• Links and images: broken internal and external links, links to redirects, empty or generic anchor text, JavaScript-only links, too many links; images with no alt text, broken images, missing dimensions, legacy formats, a lazy-loaded main image.
• Mobile, speed, security and international: a missing or zoom-blocking viewport, fixed-width layouts; HTML over 2 MB, render-blocking scripts, too many resources, no cache headers; mixed content; six hreflang checks; URL hygiene such as length, uppercase, underscores and session IDs.

Each page gets three section scores: Technical, Content Structure and SEO Readiness. A section starts at 100 and loses points for every rule that fires, weighted by severity. Once AI analysis has run, an AI visibility score joins them, and the page score becomes 30% AI visibility, 30% technical, 20% content structure and 20% SEO readiness.

The site technical score weighs every rule by its severity and by the share of pages it hits, then combines the category scores (crawlability and indexability weigh most) and deducts points if AI crawlers are blocked or llms.txt is missing. The overall AI readiness score is 25% site technical plus 75% average page score, and the dashboard prints that formula next to the number. Pages are banded by urgency: critical below 30, high from 30 to 50, medium from 50 to 70, low above 70.

A long list of failures per URL hides the pattern. So each failing check becomes one issue across the site (a missing H1 on 14 pages is one issue that reaches 14 pages), and issues are grouped into the seven root causes below, plus an "other" group.

An issue's impact grows with its severity and with how many pages it reaches; a cluster's impact is the sum of its issues. Clusters are ranked by impact, highest first, and each one leads with the single fix that removes the most of it.

• AI assistants can't fully read your site.
• Pages search engines can't reach or index.
• Missing or broken structured data.
• Titles, descriptions and headings.
• Duplicate URLs and broken links.
• Speed, mobile and rendering.
• Security and language targeting.

AI analysis is a separate pass that only runs when you ask for it. It reads each page and returns an AI visibility score from 0 to 100; ratings (excellent, good, moderate or poor) for content clarity, information completeness, brand consistency and AI discoverability; and the page's key strengths and weaknesses.

When brand context exists, it also judges relevance: whether the page is on brand, needs an update or is off brand, what matches and what's missing. AI analysis uses credits, and the dashboard shows an estimate before you start.

There are three ways to update an audit. Every page shows when it was last checked, the audit list shows progress while a run is going, and you get a notice in the app when it finishes.

• Refresh: re-crawls the site, evaluates only new or returning pages, and marks pages that have disappeared as no longer found. Pages you added by hand are never dropped.
• Re-run all pages: re-scores every page with the checks, either on their own or together with AI analysis.
• Re-check one page: after a fix, run that page again. You can also add a page the crawl missed, by path or full URL.

Failed site checks and failed page checks become action items, each with a priority, impact, effort and a link straight to the audit or the page. Failures in AI crawler access, structured data, crawlability or indexability are marked critical.

Move items through To do, In progress, Done or Dismissed. After every refresh or re-run, an open item whose problem is no longer detected is resolved automatically; if the problem comes back, the item reopens. Items you marked done or dismissed stay as you left them.