404 Errors Best Practices: SEO, UX, and Fixes

What a 404 error means for users and search engines

404 Not Found is an HTTP client error response. Under RFC 9110, it means the origin server did not find a current representation of the requested resource, or is not willing to disclose that one exists. The status does not reveal whether the condition is temporary or permanent.

Common causes include deleted pages, changed URL structures, mistyped addresses, broken internal or external links, routing failures, and invalid parameter combinations. A normal 404 does not create a sitewide ranking penalty. Search engines need this signal to stop treating unavailable URLs as indexable resources. Google states that 4xx responses other than 429 can cause the affected content to be removed from Search.

The practical harm is usually indirect. A broken internal link interrupts navigation and wastes crawl activity. A missing page with backlinks may lose referral traffic and authority that could have flowed to an equivalent replacement. A URL in an XML sitemap sends a contradictory discovery signal. A high-traffic 404 can also damage trust and conversion performance.

A soft 404 is different. It occurs when a page displays a missing-page message, or contains little usable content, while returning 200 OK or another misleading response. Google may classify it as a soft 404. Because the server still suggests that content exists, crawlers can continue requesting it.

Choose the correct response: 301, 302, 404, or 410

Choose a response according to what happened to the resource, not according to which status appears most SEO friendly. The destination of a redirect must satisfy substantially the same intent as the original URL.

Google recommends permanent redirects for permanent moves and temporary redirects for temporary moves. It also warns against redirecting many unrelated URLs to a homepage because the result can confuse users and be interpreted as a soft 404.

For a discontinued product, redirect to a true successor only when it is a close replacement. A broad category can be appropriate if it clearly fulfills the same browsing intent, but keeping a 404 or 410 is cleaner when no meaningful substitute exists. Expired jobs, events, listings, and closed locations require the same relevance test.

Prioritize 404 errors with a value and risk framework

Do not measure success by reducing the raw 404 count to zero. Active sites naturally receive malformed URLs, obsolete external links, bot probes, and requests for resources that never existed. A useful backlog ranks URLs by recoverable value and ongoing harm.

This scoring model is a triage aid, not a search engine ranking formula. Review the highest totals manually. The decision still depends on whether a relevant destination exists. A valuable old URL should not be redirected to an unrelated page merely because it has backlinks.

Create separate work queues for broken internal links, redirect opportunities, sitemap cleanup, application defects, and harmless external requests. This prevents developers from spending release capacity on random bot-generated URLs while revenue-bearing errors remain unresolved.

How to audit and diagnose 404 errors

1. Collect candidates. Use Google Search Console indexing reports, a full internal crawl, analytics landing-page reports, backlink tools, XML sitemaps, and server or CDN logs.
2. Find the source. For each important URL, identify the linking page, anchor text, sitemap, redirect, script, navigation component, or external referrer that generated the request.
3. Verify the HTTP response. Test the URL without relying only on what a browser displays. Check the initial status, every redirect hop, and the final status.
4. Render JavaScript routes. Compare the raw response with the rendered page. A single-page application may return 200 from its application shell while rendering a client-side not-found message.
5. Reconstruct intent. Review historical analytics, backlink anchors, URL patterns, content inventories, and any available archived copy to determine what the URL represented.
6. Classify the remedy. Update a link, apply a relevant redirect, restore the page, return 404 or 410, or repair the router.
7. Retest. Confirm that internal links resolve in one request, redirects terminate at the intended 200 page, and missing resources preserve a real 404 or 410.

Screaming Frog’s documented workflow can export response codes, inlinks, source pages, and anchor text. Logs add a dimension that crawls cannot: actual request frequency. Segment logs by status, URL pattern, user agent, referrer, and host so that crawler activity is not confused with human demand.

Fix broken URLs without creating redirect debt

When a live internal link points to a missing page, update the link at its source. Do this even if a 301 is also implemented. Direct links avoid unnecessary redirect hops and reduce dependence on a redirect map that may later be removed.

For migrations, build an old-to-new URL map before launch. Match pages by intent, topic, product identity, language, and locale rather than by superficial URL similarity. Test for loops, chains, redirects to another redirect, mixed protocol destinations, and accidental returns to the old host. A permanent redirect should terminate in one hop at a healthy, indexable destination.

Do not use canonical tags as replacements for redirects or error responses. A canonical is a hint about duplicate or similar indexable pages. It does not repair a broken navigation path, and a canonical placed inside a 404 template does not turn the missing URL into a valid page.

Remove dead URLs from XML sitemaps, hreflang annotations, structured data references, image manifests, navigation, and HTML canonicals. During content consolidation or decay remediation, redirect retired pages only when the surviving page substantially absorbs their purpose. Otherwise, preserve an honest 404 or 410 and strengthen links to the remaining hub or related spokes.

Build a custom 404 page that still returns 404

A custom 404 page is a user interface, not a replacement status code. It should render with the site’s normal design while the HTTP response remains 404. Test the response after changes to a CMS, CDN, reverse proxy, or application framework because any layer can replace the intended status.

A useful template should include a plain explanation, global navigation, a homepage link, site search when search works well, and a few stable category or task links. Large ecommerce, marketplace, documentation, and publisher sites can use the requested path to suggest a limited set of relevant destinations. Clearly label these as suggestions rather than pretending the original resource still exists.

• Keep the message concise and avoid blaming the visitor.
• Preserve accessibility, keyboard navigation, mobile usability, and fast loading.
• Offer a broken-link report mechanism with the requested URL captured automatically.
• Track recovery actions such as search use, category clicks, exits, and successful onward sessions.
• Avoid an enormous list of links, aggressive promotions, or a forced redirect timer.

Do not block missing URLs in robots.txt merely to hide them. A crawler generally needs access to encounter the 404 response. Also avoid returning 200 for the template, redirecting every typo, or putting personalized information into an error page that may be cached.

Prevent and repair soft 404s, especially in JavaScript applications

Soft 404s often originate in application architecture. A server may deliver the same 200 application shell for every route, after which JavaScript decides whether content exists. Google advises single-page applications to produce meaningful status handling rather than relying on a client-rendered missing-page message.

Prefer server-side or edge routing that can determine resource existence before returning the response. If the platform cannot set a status after rendering begins, route invalid paths to a server endpoint that returns a true 404. Test direct requests, refreshed deep links, escaped fragments, uppercase variants, locale paths, trailing slash variants, invalid pagination, and deleted database records.

Other failure modes include an origin returning 404 while a CDN rewrites it to 200, a reverse proxy converting all errors into a successful branded template, and APIs returning a 200 envelope with an internal error field. API conventions can differ, but public HTML routes intended for search should communicate missing resources through the HTTP layer.

The 2025 WebSci redirection study identified 62,000 custom 404 URIs in its dataset and classified nearly half as soft 404s. This does not establish a universal web rate, but it demonstrates that status and redirect defects are common enough to justify automated tests. Add known invalid routes to release checks and alert when their status changes.

Manage 404s at enterprise and crawl-budget scale

Google’s crawl-budget guidance, updated July 22, 2026, describes 404 as a strong signal not to recrawl a URL. Soft 404s can continue consuming crawl resources because the server appears to offer content. This distinction matters most on large, rapidly changing sites with faceted navigation, inventory churn, generated parameters, or millions of historical URLs.

Use logs to identify repeated 404 patterns rather than treating every URL as an isolated ticket. A shared prefix may expose a broken navigation module, malformed pagination rule, outdated mobile path, case-sensitivity change, or parameter generator. Fixing the generating rule can remove thousands of errors without creating thousands of redirects.

Keep XML sitemaps limited to canonical URLs intended for indexing. Partition sitemaps by content type, template, locale, or update frequency so anomalies can be traced to an owner. Avoid redirect chains, preserve stable URLs where possible, and retire migration redirects only after checking residual crawler requests, backlinks, internal references, and business risk.

Monitor the percentage of verified search crawler requests that receive 404, but interpret it by source and pattern. A rising rate can signal a release defect. A stable volume of requests to ancient or invented URLs may require no action. For crawl prioritization, combine request frequency with internal link count, business value, backlink strength, and the availability of a relevant remedy.

Protect topical authority, links, and AI search eligibility

Broken URLs can disconnect a site’s topical graph. If a guide acts as a hub for related tutorials, comparisons, statistics, and implementation pages, one removed spoke may strand internal links or leave part of the user journey unanswered. During consolidation, map each retired page to the surviving resource only when that resource covers the same intent. Then update contextual links and navigation instead of relying indefinitely on redirects.

Backlink analysis should focus on recoverable relevance. Use link-intersect and backlink reports to find missing URLs with authoritative references, then restore valuable content or redirect to an equivalent asset. For external typographical links, ask the publisher to correct the URL when practical. Unlinked brand mentions may support separate outreach, but they do not justify manipulating unrelated 404s.

Original statistics pages, public datasets, migration checklists, and technical comparison assets can create natural link demand when they remain at stable URLs. If these assets must move, preserve their identity through direct redirects and updated internal citations. Do not acquire expired linked URLs merely to send visitors to unrelated commercial pages. That is a high-risk relevance mismatch, not sound error management.

Google states that AI Overviews and AI Mode rely on the same foundational technical requirements as Search. Supporting pages must be indexed and eligible to appear with snippets. There is no special 404 optimization for AI systems. The practical strategy is to keep authoritative answers accessible, technically valid, internally connected, and stable. Bing, Copilot, ChatGPT, and other answer systems vary in retrieval behavior, but a missing URL cannot serve as a dependable landing source.

Evidence boundaries, implementation plan, and KPIs

What is proven

HTTP standards define 404 and 410 as distinct responses. Google officially recommends 301 for permanent moves, 302 for temporary moves, and 404 or 410 when no replacement exists. Google also documents soft 404 processing, warns against irrelevant homepage redirects, and requires indexable, snippet-eligible pages for participation as supporting links in its AI search features.

What reflects practitioner consensus

Experienced technical SEO teams prioritize internally linked errors, valuable backlinks, traffic-bearing URLs, and sitemap conflicts before harmless historical 404s. Ahrefs gives similar prioritization guidance. Recent Reddit discussions repeat these distinctions, but those reports are anecdotal and do not establish causal ranking effects.

What remains uncertain

No public evidence establishes a universal safe number or percentage of 404s. The exact discovery and citation behavior of individual AI answer systems is also variable and partly undisclosed. Independent link-rot studies demonstrate prevalence, but their datasets should not be treated as benchmarks for every site.

A practical 30-day sequence

1. Baseline true 404s, soft 404s, internal inlinks, sitemap conflicts, backlink value, traffic, and log demand.
2. Repair shared templates and internal links before creating URL-level redirects.
3. Redirect only verified equivalents, restore uniquely valuable resources, and leave irrelevant requests as 404 or 410.
4. Correct sitemaps, canonicals, hreflang references, structured data URLs, and JavaScript routing.
5. Retest status codes and redirect paths, then compare the next complete crawl and log period.

Track broken internal links, soft 404 count, sitemap URLs returning non-200 responses, redirect chains, high-value backlink targets returning 404, 404 landing sessions, recovery click rate, successful onward sessions, and repeated crawler requests to defective URL patterns. Compare like-for-like periods and annotate releases or migrations before attributing changes to remediation.