A page returning an error is a URL that answers with a 4xx or 5xx HTTP status code instead of 200. Google does not index pages that return a 4xx, and drops URLs that keep returning 5xx, so every error URL your site still links to is a dead end for visitors and a lost page in search. In Search Console these show up as Not found (404), Server error (5xx) and related statuses in the Page indexing report.
What an error response looks like
The status code is the first line of the response, before any HTML:
curl -sI https://example.com/blog/old-post | head -1
HTTP/2 404
The two families mean different things:
- 4xx: the request failed.
404 Not Foundand410 Gonesay the page does not exist.401and403say the crawler is not allowed in.429 Too Many Requestssays the server wants it to slow down. - 5xx: the server failed.
500 Internal Server Error,502 Bad Gateway,503 Service Unavailableand504 Gateway Timeoutsay the page might exist, but the server could not produce it.
Visitors see the error page; crawlers only trust the status code.
How Google treats each error
Each family affects crawling and indexing differently:
- 404 and 410. Google does not index the URL, and an indexed URL is removed from the index. Crawling of that URL gradually slows down. Google treats every 4xx except 429 this way, so a 410 does nothing a 404 does not.
- 5xx and 429. Googlebot slows its crawl rate for the whole site to avoid overloading the server. Indexed URLs are kept for a while and only dropped if the errors persist.
- Soft 404. A page that returns
200 OKbut tells the visitor the content does not exist. Search Console reports it as Soft 404. It wastes crawl budget, because Google has to fetch the page to learn it is empty.
A few broken URLs are normal. The cost comes from errors on pages that should rank, internal links to removed pages, and server errors that slow crawling.
Why it matters for AI search
AI features in Search cite pages from the index, so a URL that returns an error cannot be cited. Server errors also slow crawling, which delays new and updated pages from reaching the index and the AI answers built on it.
Why pages start returning errors
- Deleted or renamed content still linked from menus, pages or the sitemap.
- Site migrations that changed URL patterns without mapping every old URL.
- A broken template or dependency, such as a missing field or an API timeout, that turns a whole page type into 500s.
- Server limits such as traffic spikes or rate limits that return 503 or 429 to crawlers.
- Bot protection or a firewall that returns 403 to crawlers while visitors see the page.
How to find pages returning errors
- Check one URL with
curl -sIas shown above, or in your browser's network panel. Look at the status column, not the page. - Check every page. UpCrawler flags Pages returning errors on every crawled URL with a 4xx or 5xx response, and records the status code so you can tell a missing page from a failing server at a glance.
- Check what Google sees. In Search Console, open Indexing, then Pages, and look for Not found (404), Server error (5xx), Soft 404, Blocked due to access forbidden (403) and Blocked due to other 4xx issue. The Crawl stats report under Settings shows how often Googlebot hits server errors.
How to fix it
Start with the cause, not the symptom:
- 5xx errors: fix the server. Read the error logs for the failing routes. A single broken template or slow query usually explains hundreds of URLs.
- Moved content: redirect once. Send the old URL to the closest equivalent page with a 301, and point it straight at the final URL so you do not build redirect chains.
- Removed content: let it 404. If there is no equivalent page, a 404 or 410 is the correct answer. Remove the URL from internal links and the sitemap.
- Never redirect everything to the home page. Google treats mass redirects to an unrelated page as soft 404s, and visitors lose the context they came for.
- Re-crawl to confirm the URLs now return 200, a single redirect, or an intentional 404.
Call notFound() when a record does not exist, so the route renders your not-found.tsx with a real 404 status:
import { notFound } from 'next/navigation'
export default async function Page({ params }: PageProps) {
const post = await getPost((await params).slug)
if (!post) {
notFound()
}
return <Article post={post} />
}
The status code is fixed the moment the response starts streaming. If notFound() runs after that, the page returns 200 with an automatic noindex tag, which is a soft 404. Make sure the check happens first:
- No
loading.tsxon routes that can 404. A route levelloading.tsxwraps the whole page in a Suspense boundary, so even the lookup above starts the stream as200. Wrap only the slow parts of the page in<Suspense>instead. - Prerender known pages. With
generateStaticParamsandexport const dynamicParams = false, any slug that was not generated returns a real 404. - Check in
proxy.tswhen the existence check has to run per request before rendering.
For unexpected failures, an error.tsx boundary shows a friendly message, but fix the thrown error itself rather than relying on the boundary.
- If every post suddenly returns 404, go to Settings, then Permalinks, and click Save Changes without editing anything. This flushes the rewrite rules.
- Use a redirect plugin such as Redirection to map deleted or renamed posts to their replacements. Its 404 log shows which missing URLs visitors still request.
- For 500 errors, set both
WP_DEBUGandWP_DEBUG_LOGtotrueinwp-config.phpand checkwp-content/debug.log. A recently updated plugin is the most common cause.
- Deleted products and collections return 404. Add a redirect under Content, then Menus, then URL Redirects, pointing the old path at the closest product or collection.
- When you change a product handle, Shopify offers to create the redirect for you. Keep that box ticked.
- Shopify hosts the servers, so 5xx errors are rare and usually temporary. If they persist, check apps that inject scripts or proxy pages.
Let AI fix it, page by page.
Paste this prompt into AI with your project open. In UpCrawler, every affected page comes with a prompt like this, already filled in with its URL.
Fix the handler or template behind each 5xx and restore the record behind each 4xx. If the URL should no longer exist, delete the route and remove every internal link and sitemap entry pointing at it, then let it fall through to the natural 404 — do not add a handler whose only job is to announce the absence. Never add a redirect of any kind, and never mask an error by returning 200 with an error page.
Frequently asked questions
Do 404 errors hurt my SEO?
A 404 on a page that should not exist is correct and does not count against the rest of your site. 404s hurt when they hit pages that used to rank, when your own internal links point at them, or when a broken template turns important pages into errors.
What is the difference between a 404 and a soft 404?
A 404 returns the correct status code. A soft 404 returns 200 OK with "not found" content, or redirects to an unrelated page. Google detects these and reports them as Soft 404 in Search Console. Fix them by returning a real 404 or redirecting to a genuinely equivalent page.
How do I fix "Server error (5xx)" in Search Console?
Find the failing URLs in the report, reproduce the error with curl -sI, and check your server logs for that time. Once fixed, open the issue in the Page indexing report and click Validate fix.
Share it with whoever runs your site’s SEO.
