SEO audits fail when one tool is asked to answer every question. A fast Lighthouse report cannot inventory 2,000 canonicals. A crawler cannot tell you how real users experienced layout shifts over the last month. CrUX cannot identify which shared component generated 60 duplicate titles. Professional auditing works by combining evidence at different layers.
The three tools every web developer should understand are the Chrome User Experience Report, Lighthouse, and Screaming Frog SEO Spider. The essential connector that changes how an AI agent can work with crawl evidence is Screaming Frog’s MCP server.
Tool 1: CrUX shows what real eligible Chrome users experienced
CrUX means Chrome User Experience Report. It is a public dataset built from eligible real-world Chrome experiences and reports field Web Vitals at URL or origin level when enough data exists. Its core metrics include Largest Contentful Paint, Interaction to Next Paint, and Cumulative Layout Shift. Read the official CrUX overview.
CrUX answers questions such as:
- Do real visitors generally receive the primary content quickly?
- Does interaction remain responsive on actual devices?
- Do users experience unexpected visual movement?
- Is mobile experience weaker than desktop?
- Did field performance improve after enough post-release data accumulated?
This is different from running a test on one laptop. Field data includes diversity: devices, networks, cache states, locations, and browsing conditions. The CrUX API uses a rolling 28-day aggregate and updates daily, so a deployment does not instantly replace the historic distribution. Review the CrUX API model.
What to look for in CrUX
Focus on distributions and the 75th percentile rather than a single average. Look separately at mobile and desktop. Investigate whether poor LCP is widespread, whether INP suggests slow interactions, and whether CLS indicates unstable layout.
Use origin-level data carefully. It can reveal a broad site problem but can hide variation between templates. URL-level data is more precise but may be unavailable when a page lacks sufficient eligible traffic.
What CrUX cannot tell you
- Why a specific JavaScript bundle is slow.
- Which DOM element caused every layout shift.
- Whether metadata is missing.
- Whether links are broken.
- Whether a low-traffic page is fast.
- What changed five minutes after deployment.
CrUX may have no data for a new or low-traffic site. That absence does not mean the site is fast or slow; it means the public dataset cannot report a statistically eligible sample. Chrome documents that pages and origins must be publicly discoverable and sufficiently popular. See the CrUX methodology.
Tool 2: Lighthouse gives controlled lab diagnostics
Lighthouse loads a URL under a controlled test configuration and runs audits for performance, accessibility, best practices, SEO, and other web-platform qualities. It can run in Chrome DevTools, PageSpeed Insights, the command line, or a Node integration. See Chrome’s Lighthouse documentation.
Lighthouse is excellent during development because it is fast, reproducible enough for regression work, and highly actionable. A performance report can identify render-blocking work, oversized media, unused JavaScript, long main-thread tasks, layout instability, and opportunities around caching or resource delivery. Accessibility checks can expose missing names, low contrast, invalid ARIA, and structural mistakes. SEO checks can identify basic crawlability and document problems.
What to look for in Lighthouse
- Which element was selected as LCP and why it arrives late.
- Whether layout shifts come from media without dimensions, ads, fonts, or injected UI.
- Whether long tasks or heavy scripts delay responsiveness.
- Whether third-party code blocks or competes with primary content.
- Whether controls have accessible names and semantic roles.
- Whether the page returns a successful response and is indexable.
- Whether mobile layout and tap targets remain usable.
Run several tests rather than trusting one score. Keep device, throttling, location, authentication, and cache conditions consistent when comparing. Test representative templates, not only the homepage.
What Lighthouse cannot tell you
A Lighthouse score is a lab snapshot, not the experience of every visitor. Scores vary with machine load, network simulation, extensions, and page state. A score of 100 in the SEO category does not prove that titles are unique across the site, that hreflang pairs return correctly, that Search Console indexed the page, or that content deserves to rank.
Use Lighthouse to diagnose and prevent regressions. Use CrUX to understand aggregated real-user outcomes when data exists.
Tool 3: Screaming Frog reveals site-wide technical patterns
Screaming Frog SEO Spider crawls websites and organizes URL-level evidence. Spider Mode starts from a URL and follows discoverable links. List Mode requests an exact submitted set such as sitemap URLs, redirect inventories, or pages changed in a release. The free edition supports up to 500 crawled URLs; paid licensing removes that limit and enables advanced features. Read the official getting-started guide.
Screaming Frog can expose:
- 3xx, 4xx, and 5xx responses.
- Redirect chains and loops.
- Missing, duplicated, short, or long titles and descriptions.
- Missing or repeated headings.
- Canonical conflicts and non-200 canonical targets.
- Indexing directives.
- Broken internal and external links.
- Crawl depth, orphan candidates, and weak architecture.
- Missing image alt text.
- Duplicate and near-duplicate pages.
- Hreflang errors and missing return links.
- Structured-data problems when the correct validation is configured.
- Differences between raw HTML and JavaScript-rendered output.
What to look for after a crawl
Start with critical access and indexability issues: server errors, blocked pages, unexpected noindex, canonical targets that fail, and important orphan pages. Then inspect duplication, language relationships, internal links, and template patterns.
Do not treat every warning as equally important. A slightly long title on a low-priority page may be less urgent than one canonical bug affecting an entire product family. Group issues by template, route generator, component, data source, and business impact.
Spider Mode and List Mode answer different questions
Spider Mode asks, “What can be discovered through this architecture?” List Mode asks, “What is true for this exact URL set?” Use both.
In the DJAI Academy audit, a normal crawl hit the free limit because resources counted toward 500 URLs. We then used List Mode with all 205 sitemap URLs. This produced complete submitted-page coverage and confirmed 205 successful indexable pages with clean core metadata after remediation.
What Screaming Frog cannot prove
- That Google indexed or ranked a URL.
- That users had good field performance.
- That content satisfies intent.
- That every JavaScript interaction works.
- That schema is valid when validation was not enabled.
- That a warning caused a traffic change.
Crawler configuration is part of the evidence. State whether rendering, APIs, structured-data validation, robots behavior, and crawl limits were used.
The essential MCP: connect Screaming Frog to an AI agent
MCP, or Model Context Protocol, gives a compatible AI client a structured way to call tools and receive results. Screaming Frog’s MCP server exposes crawl operations, reports, exports, crawl databases, progress, and related datasets. It can also provide optional Node.js and controlled filesystem tools. The feature arrived in the version 24 release family and requires a paid licence and database storage mode. Read Screaming Frog’s official MCP guide.
Instead of manually exporting multiple CSV files and explaining every column, a user can ask an agent to:
- Crawl a domain and summarize critical issues by priority.
- Find canonicals pointing to non-200 URLs.
- Export every source page linking to a broken destination.
- Group duplicate titles by page template.
- Find English URLs without reciprocal Thai hreflang.
- Compare the latest crawl with the pre-release baseline.
- Re-crawl a corrected URL list and report what remains.
- Process large exports outside the conversation and save a concise report.
Why MCP changes the SEO game for AI agents
Without tool access, an agent reasons from screenshots, copied rows, or partial exports. It may miss scope and relationships. With MCP, it can query the underlying crawl dataset, move from a summary to exact affected URLs, and repeat the audit after a fix.
The strongest workflow is a closed verification loop:
- Agent runs or loads the crawl.
- Agent identifies patterns and maps them to likely source components.
- Human reviews business priority and risk.
- Authorized fixes are implemented.
- Build and deployment complete.
- Agent re-crawls affected URLs.
- Full List or Spider validation confirms the wider state.
This does not make the agent an autonomous SEO authority. Screaming Frog explicitly advises using AI to improve efficiency, not replace experienced analysis. Context, intent, prioritization, and final responsibility remain human concerns.
MCP security and operational cautions
Optional Node.js execution and filesystem access are powerful. Only enable them for trusted models and controlled directories. Audit data can contain private staging URLs, parameters, personal information, or credentials accidentally embedded in pages. Follow least privilege.
Large crawl datasets can also exceed an LLM context window. Prefer targeted queries, saved exports, and scripts that aggregate data outside the conversation. Start a clean task context for each audit objective and preserve an auditable record of configuration and outputs.
How to combine all four capabilities
Use them as a stack rather than competitors:
- Screaming Frog finds site-wide technical patterns and exact affected URLs.
- Lighthouse diagnoses representative pages under controlled lab conditions.
- CrUX shows aggregated real-user experience when enough eligible data exists.
- Screaming Frog MCP helps an AI agent orchestrate, query, compare, and report crawl evidence.
A practical release cycle is:
- Crawl staging with Screaming Frog.
- Run Lighthouse on each representative template.
- Fix source-level problems and rerun both.
- Deploy and inspect production HTML.
- Crawl production in Spider and sitemap List modes.
- Use MCP to compare, group, and export remaining issues when licensed.
- Inspect indexing in Search Console.
- Watch CrUX over subsequent weeks for field impact.
- Connect technical improvements to engagement and conversions.
Final rule: match the tool to the question
Ask CrUX what happened to real eligible users. Ask Lighthouse why a controlled page may be slow, inaccessible, or technically weak. Ask Screaming Frog what patterns exist across the site. Use the MCP when an AI agent needs direct, repeatable access to crawl evidence.
The SEO game changes when teams stop chasing a single score and start building an evidence system. Field data, lab diagnostics, crawl coverage, and agent orchestration each expose a different truth. Together they turn SEO auditing from a periodic spreadsheet exercise into a repeatable engineering loop.