SEO is often reduced to keywords, titles, and a plugin score. That view misses the part web developers control most directly: whether a search system can discover a URL, receive useful HTML, understand its purpose, select the correct canonical, connect its language versions, and send a visitor to a working experience.
This tutorial documents what we learned while building, auditing, repairing, deploying, and re-crawling DJAI Academy. The production sitemap contained 205 pages across bilingual tools, articles, product pages, portfolios, courses, and multiple web applications. A browser made many pages look correct. A crawler showed where the underlying contract was incomplete.
The final audit reached a strong technical SEO baseline: 205 of 205 submitted URLs returned HTTP 200 and were indexable, with no missing or duplicate titles, descriptions, H1s, or canonicals; no missing hreflang return links or x-default annotations; and no sitemap URLs returning 4xx or 5xx errors. That is meaningful engineering evidence. It is not a promise of rankings, indexing, traffic, or AI citations.
SEO is a software-quality discipline
A search engine generally moves through discovery, crawling, rendering, indexing, ranking, and presentation. Failure at an early stage makes later optimization irrelevant.
- Not discovered usually points to internal-link, sitemap, or external-discovery problems.
- Discovered but not crawled can indicate access rules, server errors, or crawl limitations.
- Crawled but not indexed can involve duplication, canonical conflict, low value, or indexing directives.
- Indexed but not ranking can involve relevance, usefulness, authority, competition, or experience.
- Ranking but not clicked can involve the title, snippet, brand trust, or mismatch with intent.
Developers shape this pipeline through HTTP responses, HTML, routing, redirects, metadata, JavaScript behavior, performance, accessibility, and deployment. SEO is therefore not a decoration applied after development. It is part of the product contract.
Start with search intent, not a list of keywords
An indexable URL should deliver a distinct workflow, input/output combination, preset, or user outcome. A PNG-to-PDF page and a PDF-to-PNG page solve opposite tasks and deserve separate URLs. A delete-PDF-pages tool and a reorder-PDF-pages tool expose different operations. A Wi-Fi QR generator needs network name, password, and security fields; a vCard QR generator needs contact fields.
By contrast, JPG and JPEG are the same format. Word-to-PDF and DOCX-to-PDF usually describe the same workflow. Creating separate pages whose only difference is a swapped keyword creates maintenance cost without creating user value.
For every proposed page, ask:
- Does it solve a different task?
- Does its interface open in the state promised by the title?
- Does its explanatory content describe real inputs, outputs, limitations, privacy behavior, and troubleshooting?
- Can the user finish the task on this URL?
- Would this page still deserve to exist if search engines sent no traffic?
This intent-first rule prevents keyword cannibalization, weak templates, doorway pages, and scaled content that exists mainly to manipulate search systems. Google’s current guidance for generative search also warns against producing a page for every query variation. Useful, differentiated content remains the durable strategy. Read Google’s generative AI optimization guide.
The indexable-page contract
Every intended indexable page should normally provide:
- A final HTTP 200 response.
- Useful main content in the initial HTML when practical.
- One clear, visible H1 aligned with the page purpose.
- A unique descriptive title.
- A useful meta description.
- An absolute self-referencing canonical.
- Crawlable internal links from a relevant hub.
- Correct index directives.
- Accurate language relationships.
- Meaningful alternative text for informative images.
- Mobile-friendly, accessible controls.
- A sitemap entry only after the production URL works.
This is a contract, not a ranking formula. A page can satisfy every technical item and still fail if it is unhelpful, unoriginal, misleading, or weaker than competing results.
Common mistakes and their SEO consequences
Missing or duplicate titles
Titles help distinguish pages and influence how a result is presented. Repeated titles make multiple URLs look interchangeable. A vague title hides the page’s actual purpose, and search engines may rewrite it. Fix the shared template or data source rather than editing hundreds of generated files individually.
Weak meta descriptions
A meta description is not a guaranteed ranking boost, and a search engine may generate another snippet. It still gives the page a concise, query-relevant summary. Missing or copied descriptions reduce publisher control and can make results less compelling.
Missing, duplicate, or client-only H1 content
A browser can display a heading after JavaScript runs while an HTML-only client receives an almost empty document. A clear server-delivered H1 and introductory copy improve semantic structure, accessibility, and crawler compatibility. Multiple H1s are not automatically fatal, but one unambiguous primary heading is an excellent engineering convention.
Canonical conflicts
Canonical signals tell search systems which URL represents duplicate or highly similar content. Problems arise when HTML points to one URL, the sitemap lists another, internal links use a third, and redirects land on a fourth. Signals become diluted and the wrong URL may be selected.
Use a self-referencing canonical on the preferred page, link internally to that URL, list it in the sitemap, and permanently redirect retired duplicates where appropriate. Do not use robots.txt as a canonicalization tool. Google describes redirects and canonical link annotations as strong signals, while sitemap inclusion is weaker. Review Google’s canonical documentation.
Broken or non-reciprocal hreflang
Multilingual pages form a relationship, not a one-way pointer. If English declares Thai as an alternate, Thai should return the relationship. Each language page should canonicalize to itself, and every alternate should resolve to a final indexable page. The x-default annotation identifies the fallback experience when no language match is selected.
Incorrect mappings can cause search systems to ignore the cluster or show an unexpected language. More importantly, they usually reveal that translation data, routing, and metadata generation disagree.
Broken links, redirect chains, and orphan pages
Broken links create dead ends. Redirect chains waste requests and slow navigation. Orphan pages may exist in a sitemap without receiving contextual links from the site. A sitemap supports discovery, but it does not replace architecture. Link every valuable page from a relevant category hub and connect it to genuinely related pages.
Empty alt text on meaningful images
Empty alt text is correct for decorative media. It is wrong for a product screenshot, diagram, or meaningful interface image when that image communicates information. Descriptive alt text supports accessibility and gives image systems clearer context. Avoid keyword stuffing; describe the image’s role.
Misleading structured data
Structured data should describe visible reality. Never invent reviews, ratings, prices, availability, credentials, or features. Valid JSON-LD does not guarantee a rich result, and schema cannot rescue thin content. Use it to clarify entities and relationships such as Organization, Article, SoftwareApplication, Course, BreadcrumbList, or an appropriate HowTo.
Treating a score as proof of SEO success
A Lighthouse SEO score covers a useful technical subset. It does not measure search demand, content originality, backlinks, index coverage, intent satisfaction, or conversions. Likewise, a clean crawl does not prove that Google indexed every page. Use Search Console for indexing and search performance rather than relying on a site: query, whose results are subject to retrieval limits.
What an SEO crawler does
An SEO crawler visits URLs, follows links, reads responses, and builds a site-wide dataset. Instead of manually inspecting one browser tab at a time, developers can compare hundreds of pages across consistent fields:
- Status codes and redirect targets.
- Titles, descriptions, headings, and word counts.
- Canonicals and robots directives.
- Internal links, anchor text, crawl depth, and orphan candidates.
- Image URLs and alt text.
- Hreflang annotations and return links.
- Duplicate or near-duplicate content.
- Structured data when validation is enabled.
- Performance data when relevant APIs are connected.
A crawler turns isolated symptoms into patterns. If 40 pages share a missing title, the likely cause is a template. If only three legacy blog posts have broken language links, the likely cause is stored data. It also creates a repeatable baseline: crawl before a fix, crawl after deployment, and compare.
How we used Screaming Frog on DJAI Academy
We ran Screaming Frog SEO Spider 24.3 locally against production. The initial standard Spider crawl reached the free edition’s 500-URL limit because pages, scripts, images, and other resources all counted. Even that partial crawl exposed four classes of issue.
- Two client-rendered bio pages lacked crawlable H1 content in the initial response.
- Three legacy blog translation pairs lacked reciprocal hreflang.
- One QR page family was missing x-default.
- Meaningful product images had empty alt text.
We corrected the source implementations and rebuilt the complete deployment. Production then exposed a subtler issue: its persistent blog file used an older data shape than the repository seed. English metadata could not always resolve the real Thai partner. The solution had to support legacy production data rather than assuming the newest schema.
The first compatibility fix introduced another asynchronous lookup. Next.js produced the metadata, but standard HTML crawling could receive the canonical and language annotations too late because they were streamed. We replaced that dependency with a synchronous seeded lookup so critical metadata appeared in the initial document head.
That episode produced one of the strongest lessons in this project:
Framework-generated metadata is not verified until you inspect the actual deployed response and the client type that must consume it.
Finally, we downloaded the 205 live sitemap URLs and used List Mode. This focused the free allowance on the authoritative submitted-page set instead of spending it on every discovered resource. Spider Mode told us what the architecture exposed; List Mode told us whether every submitted URL met the release contract.
Screaming Frog MCP and what we actually used
Screaming Frog version 24 introduced an MCP server that lets compatible AI assistants start crawls, inspect progress, query crawl data, run analysis, and create exports through natural-language tool calls. It is a paid-licence feature, requires database storage mode, and can optionally expose Node.js and controlled filesystem capabilities. Read the official Screaming Frog MCP configuration guide.
For this DJAI audit, we used the local CLI, not the MCP server. The free edition and List Mode were sufficient for the measured 205-page verification. A truthful case study must preserve that distinction.
The MCP becomes valuable for repeatable agent workflows. An agent can be asked to crawl production, export non-200 canonical targets, group missing headings by template, find English pages without Thai return links, compare two crawl databases, and re-crawl corrected URLs. The human still owns priorities, business context, security, and final judgment. Screaming Frog itself recommends AI as an efficiency tool rather than a replacement for experienced SEO analysis.
AI search optimization begins with conventional SEO
Google states that its generative search experiences remain grounded in core Search indexing and ranking systems. That means the foundation is familiar: crawlable content, correct canonicalization, internal links, useful original information, clear entities, and a satisfying page.
Content becomes easier to retrieve and cite when it:
- Answers the main question early.
- Uses descriptive headings.
- Defines technical terms directly.
- Includes firsthand examples, measurements, and limitations.
- Identifies the author, publisher, product, and organization consistently.
- Supports explanations with relevant diagrams or media.
- Links to authoritative sources for time-sensitive facts.
- Maintains accurate structured data that matches visible content.
DJAI also added llms.txt with canonical organization, product, course, community, and tool links. This should be described accurately: llms.txt is an emerging proposal, not a universal standard or ranking mechanism. It complements rather than replaces HTML, robots.txt, XML sitemaps, structured data, and normal internal linking. See the llms.txt proposal.
Avoid so-called GEO tricks such as mass-producing Q-and-A pages, inventing citations, stuffing every heading with a query, or publishing unreviewed AI text. AI systems still need trustworthy source material. Better retrieval begins with better publishing.
Performance and accessibility belong in the same review
Representative DJAI mobile pages scored 98 to 100 for Lighthouse performance, 100 for accessibility, and 100 for the Lighthouse SEO category. We repaired missing accessible names, tested mobile layouts at 390 pixels, prevented horizontal overflow, and changed noncritical AdSense loading from beforeInteractive to afterInteractive.
These changes help real users first. Semantic headings, accessible controls, stable layouts, fast primary content, and nonblocking scripts also make pages easier to process and less frustrating to use. Still, Lighthouse is lab evidence. Real-user performance requires field data such as CrUX when enough eligible traffic exists.
The release workflow every developer should adopt
- Define a distinct intent and useful outcome.
- Build a complete working page.
- Test the application and production build.
- Inspect raw HTML and rendered behavior.
- Crawl local or staging output.
- Verify redirects, canonicals, hreflang, robots, links, and sitemap entries.
- Deploy through the authorized release path.
- Inspect representative production responses.
- Run a production Spider crawl.
- Run a sitemap List crawl for exact coverage.
- Fix root causes, rebuild, redeploy, and re-crawl.
- Submit the sitemap and inspect representative URLs in Search Console.
- Monitor impressions, clicks, landing-page engagement, conversions, indexing, and field performance.
SEO fixes are not complete when code changes. They are complete when the deployed response has been independently verified.
Final lessons
The DJAI audit did more than remove warnings. It showed that SEO metadata is application data, legacy production state can break language relationships, JavaScript timing changes what crawlers receive, and crawler configuration changes what an audit proves.
The final clean crawl established a highly optimized technical SEO baseline across the audited sitemap. It did not guarantee rankings. That boundary is healthy: engineering should produce reliable evidence, content should produce genuine value, and Search Console should reveal how real search systems respond over time.
Build for people. Make the meaning explicit for machines. Verify through the eyes of a crawler. Then measure what happens in the real world.