OPEN-SOURCE SEO CRAWLER + MCP FOR AI AGENTS

SEO Screaming Toad: an open-source SEO crawler built for AI workflows

Audit technical SEO, JavaScript SEO, canonicals, hreflang, sitemaps, structured data, and internal links with local evidence—then give AI agents 23 bounded MCP tools to inspect and compare real crawl results.

Open the English repository Read the installation guide

Free under the MIT License · Local-first data · Current release candidate · Not affiliated with or endorsed by Screaming Frog Ltd.

🐸

SEO Screaming ToadDJAI Toad · local-first

MIT

CrawlerMCPEvidence

AUD-04Canonical signalswarning

AUD-09Hreflang reciprocitywarning

AUD-13Structured dataerror

crawl_status → completedissue_explain → evidencecrawl_compare → fixed

23MCP tools for AI agents

13versioned technical SEO families

Raw + renderedseparate HTML and JavaScript evidence

Local-firstSQLite/WAL on your machine

What is SEO Screaming Toad?

SEO Screaming Toad, also called DJAI Toad, is an open-source SEO crawler and website audit tool for sites you own or are authorized to test. It discovers pages and resources, audits technical SEO signals, and stores URL inventory, page evidence, link relationships, and findings in SQLite for review, comparison, and export.

Its defining feature is a purpose-built MCP server for AI agents. Instead of giving an agent an unrestricted shell or HTTP client, Toad exposes bounded tools for profiles, scope previews, crawl control, issues, page evidence, comparisons, and managed reports.

CORE CAPABILITIES

From crawl discovery to evidence you can act on

This is more than a list of warnings. The workflow connects guarded URL discovery, extraction, versioned rules, stored evidence, and a comparable recrawl after changes.

Guarded crawler

Concurrent Go crawling with robots.txt, sitemap discovery, URL normalization, deduplication, TLS, redirect validation, and per-host politeness.

Raw + JavaScript audits

Keep server HTML separate from optional Playwright-rendered evidence so client-side changes remain visible.

Evidence-backed findings

Each finding retains a rule ID, version, severity, subject, evidence, remediation, and explicit limitation.

Canonical and indexability

Check missing, conflicting, invalid, chained, failed, and noindex canonical targets plus robots directives.

International SEO

Check hreflang codes, reachable targets, and reciprocal language relationships.

Content and metadata

Inspect titles, descriptions, H1s, exact duplicates, near-duplicate signals, and configurable length thresholds.

Link architecture

Review inlinks, outlinks, crawl depth, broken internal targets, nofollow observations, and orphan-like pages.

Structured data

Detect JSON-LD syntax and basic structural problems while disclosing vocabulary and rich-result limitations.

Recoverable campaigns

Pause, resume, cancel, timeline, durable frontier, checkpoints, and recovery for longer-running work.

Reports and comparison

Export CSV, NDJSON, or XLSX and compare added, removed, changed, new, and fixed results.

Agent-ready interface

Operate the same stored evidence through the dashboard, JSON CLI, local API, or 23-tool MCP server.

Actually open source

The crawler, dashboard, CLI, local API, reports, and MCP server are available under the MIT License.

Important SEO audit coverage

The current engine has 13 versioned audit families. Findings are technical observations, not guarantees of indexing or rankings.

Area	What it checks	What to remember
Responses	Failed pages, redirects, and broken internal targets	Failures can be temporary
Metadata	Missing, short, long, or duplicate titles and descriptions	Length thresholds are editorial guidance
Canonicals	Missing, conflicting, invalid, chained, failed, or noindex targets	Canonicals are hints
Indexability	Non-200, noindex, robots, and sitemap coverage	Search engines apply other policies
International	Hreflang codes, targets, and reciprocity	Market intent needs human review
Architecture	Depth, inlinks, outlinks, nofollow, and orphan-like pages	Some utility pages are intentionally isolated
Media	Missing image alt attributes and failing image resources	A crawler cannot infer visual intent
Structured data	JSON-LD syntax and structural consistency	It does not guarantee a Google rich result

How the crawler turns a site into SEO evidence

Define scope

Set seed URLs, allowed hosts, exclusions, URL ceiling, depth, rate, and raw or rendered mode.

Preview before fetching

Normalize candidate URLs and explain whether the stored profile includes them.

Crawl politely

Enforce robots, DNS/IP guards, TLS, redirects, response limits, and per-host delay.

Extract and audit

Store metadata, links, images, hreflang, structured data, and versioned findings.

Fix and recrawl

Compare added, removed, changed, new-issue, and fixed-issue results between runs.

How to install and use the SEO crawler

The project is currently a release candidate and does not yet publish a signed stable release. The most transparent route is to clone the repository, inspect the source, and build locally.

Use the Go version in .go-version. Node.js and pnpm are needed to build the dashboard or optional JavaScript renderer.

Open http://127.0.0.1:7331.
Create a project for the site or client.
Create a profile with a conservative URL ceiling.
Preview scope and verify included paths and subdomains.
Start the audit and monitor discovered, queued, fetched, analysed, and failed counts.
Read the summary, issue explanations, and page evidence before changing the site.
Recrawl with a comparable configuration to verify the outcome.

terminal

git clone https://github.com/lovecatisgood-sudo/Free-Opensource-SEO-Screaming-Toad-not-Frog-tool-with-100million-url-crawl-potential.git
cd Free-Opensource-SEO-Screaming-Toad-not-Frog-tool-with-100million-url-crawl-potential
make bootstrap
go run ./cmd/seo-auditor

Open the English repository

MCP FOR AI SEO

Give an AI agent an SEO crawler—not your whole shell

The MCP server communicates over stdio and calls Toad's authenticated loopback API. Agents can operate real SEO workflows without receiving a generic shell, arbitrary SQL, filesystem access, a browser primitive, or unrestricted HTTP fetching.

1AI agent / MCP client

2seo-auditor-mcp (stdio)

3Authenticated local API

4Guarded crawler + SQLite evidence

Example MCP configuration

{
  "mcpServers": {
    "seo-screaming-toad": {
      "command": "/absolute/path/to/seo-auditor-mcp",
      "env": {
        "SEO_AUDITOR_BIND_HOST": "127.0.0.1",
        "SEO_AUDITOR_BIND_PORT": "7331"
      }
    }
  }
}

Projects and profiles

project_createproject_listprofile_createprofile_listcrawl_preview_scope

Crawl lifecycle

crawl_startcrawl_statuscrawl_pausecrawl_resumecrawl_cancelcrawl_listcrawl_timeline

Audit evidence

audit_summaryissue_listissue_explainpage_listpage_getlink_listcrawl_compare

Reports and diagnostics

report_exportdiagnostic_createartifact_get

Recommended AI-agent workflow

Call health_get to verify MCP and API readiness.
Read project_list and profile_list before creating state.
Use crawl_preview_scope and disclose limits and exclusions.
Call crawl_start with an idempotency key after operator intent is clear.
Poll crawl_status every two to five seconds until a terminal state.
Read audit_summary, then paginate issue_list and page_list.
Use issue_explain and page_get before recommending a fix.
Create report_export only when requested.

AI Search Optimization, AEO, GEO, and AI-search readiness

AI Search Optimization aims to make content and entities crawlable, understandable, attributable, and useful for answer-oriented retrieval. Related labels include AEO and GEO. ASO normally means App Store Optimization, although some people also use it for AI Search Optimization.

SEO Screaming Toad cannot guarantee an AI citation. It can audit the technical foundation used by search crawlers and retrieval systems: response status, canonicals, indexability, language relationships, internal links, structured data, raw and rendered content, and duplicate signals.

Crawlability

Important pages and links are discoverable without hidden interaction.

Entity clarity

Titles, H1s, canonicals, language, and structured data align with visible content.

Retrievable answers

Definitions, steps, limits, and evidence appear in readable HTML.

Multilingual consistency

Thai and English URLs use self-canonicals and reciprocal hreflang.

Source confidence

Authors, organizations, repositories, licensing, and limitations are explicit.

Rendered parity

Critical information does not disappear or mutate incorrectly after JavaScript runs.

Is it a Screaming Frog alternative?

SEO Screaming Toad is independent and uses 'Screaming Frog' only for descriptive comparison. It does not claim feature parity and is not affiliated with Screaming Frog Ltd. Its differentiators are open code, local evidence, a versioned finding model, and a first-class MCP interface. If you rely on mature commercial integrations, test both tools on a representative authorized site and compare coverage, false positives, rendered output, exports, and operational cost.

Area	SEO Screaming Toad	What to evaluate
License	MIT open source	Your team can inspect and adapt the source
Data	Local SQLite/WAL	You own retention and backup
AI automation	23 bounded MCP tools	The local supervisor must be running
JavaScript	Optional isolated renderer	Raw and rendered evidence stay separate
Maturity	Release candidate	Validate your real environment before migration

EVIDENCE BEFORE CLAIMS

100M URLs: theoretical architecture, not a capacity promise

The project has completed synthetic production-path campaigns at one million and five million URLs. The 5M run retained 5,000,000 pages, 4,999,999 links, and 15,884,167 findings in about 11.1 GB of SQLite storage. It did not use live networking, so it does not prove DNS, TLS, robots, redirects, bandwidth, host scheduling, or JavaScript rendering at that scale. The 100M+ design remains a segmented-campaign research direction.

100Kregression gate1Msynthetic evidence5Mone synthetic run100M+theoretical only

Security, privacy, and responsible use

Crawl only sites you own or are authorized to test.
Start with low URL and concurrency limits, then increase them with server capacity.
Do not disable TLS, robots, DNS/IP guards, or redirect validation to force a crawl through.
Databases and exports may contain client URLs or evidence; protect them with access and retention policies.
The current MCP does not accept credentials, arbitrary headers, or private-network targets.
Treat findings as evidence for human decisions, not instructions to auto-fix every warning.

Frequently asked questions about the SEO crawler and MCP

Is SEO Screaming Toad free?

Yes. The source is available under the MIT License and can be inspected, built, and run locally.

Can it replace Screaming Frog for every workflow?

That is not the claim. Toad is a release candidate and does not claim feature parity. Validate it against your sites and workflow.

How does MCP help SEO work?

MCP lets an agent start and monitor crawls, read findings, inspect page evidence, compare runs, and create reports through bounded tools.

Can a crawler improve AI Search Optimization?

It can audit crawlability, canonicals, language, structured data, links, and raw/rendered content—the technical foundation for retrieval—but cannot guarantee citations.

Does crawl data go to the cloud?

The crawler and SQLite database are local by default. Operators must still review their renderer, environment, and export workflow.

Can it crawl 100 million URLs today?

100M+ is an unverified architectural target. Current evidence is one synthetic production-path campaign at five million URLs.

Do I always need JavaScript rendering?

No. Raw mode is the faster baseline. Use rendered mode when client-side JavaScript changes important SEO evidence.

How should I start safely?

Confirm authorization, choose conservative scope and budgets, preview URLs, then inspect terminal reasons and evidence before making changes.

SEO SCREAMING TOAD

Start with an open SEO tool an AI agent can understand

Open the English repository, read the limitations, inspect the source, and begin with a small authorized crawl. Use stored evidence and comparable recrawls to improve technical SEO without guessing.

Open the English source code Browse all free DJAI tools

COMPLETE TOOL DIRECTORY