DJAI announces SEO Screaming Toad
DJAI Academy and the DJAI development community are announcing SEO Screaming Toad, also called DJAI Toad: a free, open-source technical SEO crawler created for teams that need to understand how websites are discovered, crawled, interpreted, and maintained.
The ambition is straightforward. Screaming Frog SEO Spider is one of the best-known professional website crawlers in the SEO industry. It has established the desktop-crawler workflow that developers, agencies, publishers, and technical SEO specialists use to investigate broken links, redirects, metadata, indexability, canonicals, duplicate content, hreflang, structured data, and site architecture. SEO Screaming Toad brings that class of work into an MIT-licensed, local-first product whose source code, evidence model, reports, API, dashboard, and MCP interface can be inspected and extended by the community.
This is an alternative, not an imitation claim. SEO Screaming Toad is independent, is not affiliated with or endorsed by Screaming Frog Ltd., and does not yet claim complete feature parity or superiority. Teams that depend on mature commercial integrations should validate those workflows before migrating. Our achievement is that a small AI-assisted development community has built a working release candidate that can crawl a real production site, preserve auditable evidence, explain versioned findings, export reports, compare recrawls, and let an AI agent participate through bounded tools.
That is already useful. It is also only the beginning.
Why technical crawlability matters
A website can look excellent to a human and still communicate poorly with a crawler. Navigation may hide important pages. A canonical can point to the wrong URL. A filter page can accidentally become indexable. Two tools with different purposes can contain so much shared template copy that their unique value becomes difficult to recognize. A JavaScript application can show one title to a browser while returning different raw HTML to a crawler.
Technical SEO does not guarantee rankings. It removes avoidable ambiguity and waste from the path between a website and the systems trying to understand it.
A capable SEO crawler helps answer practical questions:
- Which URLs can actually be discovered through links?
- Which responses are successful, redirected, blocked, or broken?
- Does each important page have a clear title, description, heading, canonical, and language identity?
- Do Thai and English pages point back to each other correctly?
- Are noindex directives intentional or accidental?
- Are sitemap entries healthy and aligned with canonical URLs?
- Which pages look exact or nearly duplicated?
- Can developers retain evidence and prove whether a recrawl improved the result?
The official Screaming Frog SEO Spider addresses many of these established audit tasks. SEO Screaming Toad enters the same field with a different product philosophy: open code, local evidence, explicit rule limitations, and agent-assisted workflows.
What DJAI built
SEO Screaming Toad combines several systems that would normally be separate engineering projects.
| Layer | What it contributes |
|---|---|
| Guarded crawler | Discovers authorized public URLs while enforcing robots policy, scope, DNS and IP safety, redirects, response budgets, retries, and host politeness |
| Evidence store | Retains pages, links, raw and rendered extraction, findings, lifecycle events, and comparisons in local SQLite/WAL storage |
| Audit engine | Runs versioned technical SEO rules and stores severity, classification, evidence source, remediation, and limitations |
| Dashboard | Provides crawl profiles, live status, searchable page inventory, findings, explanations, history, and XLSX export |
| CLI and local API | Support repeatable automation without turning the crawler into an unrestricted remote service |
| MCP server | Gives AI agents 27 bounded tools for projects, profiles, scope previews, crawl control, custom audits, evidence, comparisons, and managed reports |
The distinction between evidence and recommendation is central. A warning is not automatically a mistake. An error label is not automatically a business priority. A title above an editorial length threshold is not proof that Google will rank it badly. A noindex page may be an accidental loss of search visibility, or it may be the correct duplicate-control policy for an interactive filter.
SEO Screaming Toad keeps the observed signal available so a developer can make that decision with context.
The real DJAI Academy audit
We did not announce the crawler using a fabricated demo. We used it against DJAI Academy's own production website, a bilingual site with course pages, development services, a blog, many browser tools, a mobile-app section, and several connected product surfaces.
The documented crawl ran on 30 July 2026 using SEO Screaming Toad 2.0.0-rc.4. It used raw HTML mode, stayed on the exact www.djai.academy host, respected robots.txt, excluded subdomains, used a 100,000-URL safety ceiling, and disabled explicit compression negotiation because the host had shown an Accept-Encoding compatibility problem.
The crawl completed when the same-host frontier emptied—not when it reached the safety ceiling.
| Audit measure | Observed result |
|---|---|
| Discovered resources | 287 |
| Fetched resources | 287 |
| Analysed HTML pages | 222 |
| Fetch failures | 0 |
| Final fetched responses | All 2xx |
| Sitemap records | 1, status OK |
| Rule-level errors | 8 |
| Warnings | 66 |
| Informational observations | 20 |
| Total technical observations | 94 |
So the accurate value for “XXX” is 94 technical observations.
It would be inaccurate to call all 94 “mistakes.” The audit contained a mixture of review items, editorial diagnostics, and information. It would also be inaccurate to say that Screaming Frog missed these observations, because this crawl was not a controlled differential test against Screaming Frog with the same configuration and version. That benchmark is on our quality roadmap. Until it is run and published reproducibly, we will not manufacture a comparison result.
What we can say is stronger than a marketing exaggeration: SEO Screaming Toad independently completed a production crawl, returned zero fetch failures, retained the page inventory, and created a prioritized review queue that drove concrete engineering decisions.
What were the high-severity findings?
There was no confirmed high-priority site failure in this raw crawl.
That is an important result, not a disappointing headline. All 287 fetched resources returned successful responses. Every analysed HTML page had a title, meta description, and canonical. The sitemap was available. We did not observe broken final responses, missing titles, missing descriptions, or missing canonicals in the accepted scope.
The audit engine did generate eight rule-level AUD-05 indexability errors. They were eight blog category-filter URLs returning HTTP 200 with noindex directives.
A superficial workflow could have removed noindex from all eight pages to make the dashboard green. That would have been the wrong fix.
Human inspection showed that these parameterized filter views used noindex, follow, canonicalized to the Thai or English blog hub, reused hub metadata, and were intentionally absent from the XML sitemap. Together, those signals were consistent with deliberate duplicate control. The useful outcome was not “remove noindex.” It was to document the product decision and keep the signals aligned.
If DJAI later decides those categories should rank as independent search landing pages, the correct change is larger: stable clean URLs, unique localized metadata, useful category copy, self-referencing canonicals, reciprocal language links, internal discovery, and sitemap inclusion after verification.
This is why severity and priority must remain separate.
The medium-priority finding we could easily have overlooked
The most useful medium-priority queue contained 24 near-duplicate similarity observations across related QR generators, image tools, PDF tools, and media converters.
Similarity is expected in a product family. Every converter shares navigation, controls, privacy explanations, and parts of a design system. The SEO question is whether each canonical URL still makes its distinct task obvious.
For example, AVIF to JPG and AVIF to PNG are related, but they do not serve exactly the same output requirement. DOCX to Markdown and DOCX to Text are related, but one preserves useful structure while the other intentionally produces plain text. If the explanatory content discusses only generic conversion, search engines and AI retrieval systems receive weak evidence about why both pages deserve to exist.
Without an automated similarity queue, a developer reviewing one page at a time may never notice how strongly the templates resemble one another across the site.
The proper response is not blind consolidation. Each pair should be classified:
- Intentional product-family similarity with sufficient unique value
- Distinct intent but insufficient page-specific explanation
- Genuinely duplicated intent that should be consolidated or redirected
The improvement work includes clearer use cases, input and output differences, format limitations, examples, privacy behavior, internal links, and task-specific FAQs. This helps conventional search and AI retrieval because the unique purpose of each page becomes easier to extract.
Lower-priority findings that still matter
The audit also recorded metadata-length and heading observations.
| Observation | Count | Interpretation |
|---|---|---|
| Titles above 60 characters | 12 | Editorial review; not a ranking violation |
| Titles below 30 characters | 3 | Check whether the page purpose is sufficiently descriptive |
| Descriptions above 160 characters | 7 | May be truncated or rewritten; prioritize clarity |
| Descriptions below 70 characters | 13 | Informational opportunity to explain value better |
| Title and H1 equality | 7 | Often acceptable; do not rewrite solely to make them different |
These thresholds are diagnostics. Search engines can rewrite snippets, and a natural 62-character title can be better than an awkward 58-character title. Toad creates the review list; the editorial team supplies judgment.
What the earlier screening helped DJAI fix
Before the documented 94-observation course crawl, an earlier raw screening fetched 285 URLs and analysed 220 HTML pages. That pass produced 103 observations, including non-reciprocal hreflang on filtered blog URLs, repeated filter metadata, metadata-length reviews, and two notable near-duplicate template pairs.
The engineering work that followed included:
- Removing hreflang from noncanonical parameter-filter views so language clusters describe indexable canonical pages instead
- Preserving noindex, canonical, and sitemap alignment for user-only filters
- Strengthening localized metadata and page-specific explanatory content where it added genuine value
- Improving crawlable internal discovery across the growing tool collection
- Separating Thai and English canonical routes and validating document language
- Adding an explicit safe compression-compatibility setting after Hostinger returned 403 responses when an Accept-Encoding header was sent
- Expanding automated deployment checks so canonical, hreflang, sitemap, content, and internal-link regressions fail before release
The two crawls used evolving rules and slightly different scopes, so the difference between 103 and 94 is not a scientific before-and-after score. We do not present it as one. The value is the repeatable workflow: observe, inspect evidence, decide, fix the source, deploy, and recrawl.
A crawler built for AI-agent collaboration
SEO work frequently fails at the handoff. An auditor exports a spreadsheet, a developer receives a vague ticket, and nobody can easily verify whether the fix changed the underlying evidence.
SEO Screaming Toad's MCP server is designed to shorten that loop. An MCP-compatible agent can use 27 bounded tools to create projects, define crawl profiles, preview scope, start and monitor authorized crawls, inspect pages and links, explain findings, run bounded custom audits, compare compatible crawls, and generate managed reports.
The agent does not receive a generic shell, arbitrary SQL, unrestricted filesystem access, or an unrestricted web-fetch primitive through MCP. It operates the same guarded crawler and stored evidence used by the dashboard.
That makes workflows like this possible:
AI does not replace technical judgment. It makes the evidence easier to interrogate and the verification loop easier to repeat.
SEO, crawlability, and AI Search Optimization
AI Search Optimization, Answer Engine Optimization, and Generative Engine Optimization are evolving labels. The technical foundation remains familiar: a system cannot reliably retrieve content that is blocked, contradictory, duplicated without context, weakly linked, or absent from the initial response.
SEO Screaming Toad can help inspect that foundation:
- Response status and crawl access
- Canonical and indexability alignment
- Reciprocal language relationships
- Internal links and crawl depth
- Raw versus rendered content
- Structured-data syntax and vocabulary
- Duplicate and near-duplicate signals
- Sitemap and robots relationships
No crawler can guarantee indexing, rankings, traffic, or citation by an AI system. The value is making the site easier to discover, less ambiguous to interpret, and easier for humans to maintain.
Our roadmap: quality before exaggerated scale
SEO Screaming Toad is currently a release candidate. DJAI's roadmap is continuous and quality-led. We want the product to earn standing in the field through reproducible evidence, transparent limitations, useful rules, and community contribution.
The active roadmap includes:
- Deeper Schema.org vocabulary checks and separately versioned search-feature profiles
- A bounded custom audit framework for selectors, extraction, and reusable checks
- Reproducible Lighthouse lab evidence, accessibility checks, and separately labelled PageSpeed or CrUX field data
- Least-privilege Search Console and analytics enrichment
- Stronger content differentiation, internal-link scoring, architecture views, and explainable orphan candidates
- Stored rendered DOM, screenshots, console diagnostics, and safe authenticated crawling under explicit budgets
- Clean-machine testing across Linux, macOS, and Windows
- A public conformance fixture suite measuring expected findings, expected absences, precision, recall, and rule-family coverage
- An authorized, same-scope differential benchmark against Screaming Frog SEO Spider 24.3
The project includes architecture for segmented campaigns beyond 100 million URLs, but that remains a theoretical direction—not a verified live-network capacity or support promise. Current published production-path evidence includes synthetic campaigns up to five million URLs. Audit quality and URL capacity are different measurements, and we will continue reporting them separately.
An achievement built in public
SEO Screaming Toad demonstrates what AI-assisted software development can achieve when speed is paired with architecture, security boundaries, tests, evidence, and honest product communication.
It is not finished. That is part of the announcement.
DJAI Academy trainers, developers, learners, and the Siamese Cat Dev creator community now have a real platform to improve together. Contributors can inspect the code, propose rules, improve platform support, refine reports, test crawl behavior, and challenge our assumptions.
Explore the English SEO Screaming Toad product guide, read the open-source repository, or open the Thai product guide.
If your team evaluates Toad against another crawler, use the same authorized target, scope, rendering mode, limits, and timing. Publish the configuration with the result. That is how an open tool earns trust—and how DJAI intends to build its status in technical SEO, AI SEO, and search engineering.
Final word
The headline is not that a frog has been replaced by a toad overnight.
The headline is that an open development community built a serious alternative, used it on its own production website, collected 94 evidence-backed technical observations, made better engineering decisions, and published the limits alongside the achievement.
We will keep crawling, fixing, comparing, and improving.