A page is excluded by the noindex tag when it tells search engines not to add it to their index, either with a <meta name="robots" content="noindex"> tag or with an X-Robots-Tag: noindex HTTP header. Google respects it, so the page can be crawled but will never appear in search results or in AI Overviews. That is exactly what you want for a login screen and a disaster for a product page.
Where the noindex directive lives
There are two places, and the second one is easy to miss because it is invisible in the page source.
The robots meta tag, inside <head>:
<meta name="robots" content="noindex, nofollow">
The same rule can target Google only with <meta name="googlebot" content="noindex">, so search for both names.
The X-Robots-Tag response header, set by the server, CDN or framework:
HTTP/2 200
content-type: text/html
x-robots-tag: noindex, nofollow
Either one on its own is enough to keep the page out of the index. When the tag and the header disagree, the more restrictive rule wins.
Why this happens by accident
Almost nobody noindexes an important page on purpose. The usual causes are:
- A staging setting that reached production. A site built with
noindexon everything before launch keeps it after launch because the switch lived in shared code. - WordPress "Discourage search engines from indexing this site". The checkbox under Settings, then Reading, adds a noindex robots tag to every page.
- A per page toggle in an SEO plugin, flipped once for a draft and never flipped back.
- A header added at the edge. A CDN rule, reverse proxy or framework config that sends
X-Robots-Tagfor more paths than intended. - A shared layout that sets
robots: noindexfor one section but wraps more routes than the author realised.
We made the first mistake ourselves: UpCrawler ran with noindex on every page until launch. The fix that prevents it from happening again is to tie indexing to the environment, so production is indexable and every other environment is not, without anyone remembering to flip a switch.
How to find pages excluded by noindex
- Search Console. Open Indexing, then Pages, and look for URL marked 'noindex', the reason older reports called Excluded by 'noindex' tag. The list shows every URL Google found with the directive.
- Check one page, both places. View the source and search for
noindex, then check the response headers:
curl -sI https://example.com/pricing | grep -i x-robots-tag
- Check every page. UpCrawler flags Pages blocked from indexing on each crawled URL that carries noindex in either the meta tag or the header, and marks it critical because the impact is total.
How to fix it
Remove the directive from every page that should rank, in both places, and keep it only on routes that genuinely should stay out of search. Remove it from the server HTML, not with JavaScript after the page loads: a page that arrives with noindex may not be rendered at all, so a script that removes the tag may never run for search engines.
The robots field in metadata applies to every route below the layout that sets it, so set it once in the root layout, gated on the environment:
export const metadata: Metadata = {
robots: process.env.NEXT_PUBLIC_ENVIRONMENT === 'prod'
? { index: true, follow: true }
: { index: false, follow: false }
}
Then check next.config.ts for any X-Robots-Tag entries in headers() and scope them to the paths that should stay private, such as /app/:path*. app/robots.ts controls crawling, not indexing, so it cannot remove a noindex.
One more source is easy to miss: a page that calls notFound() after the response has started streaming returns 200 with an automatic noindex tag. If Search Console lists pages you never noindexed, check the pages returning errors guide.
Go to Settings, then Reading, and untick Discourage search engines from indexing this site. Then open the affected pages in your SEO plugin and make sure the per page setting allows indexing.
Shopify does not noindex storefront pages by default. If a page is excluded, look for a noindex robots tag added to theme.liquid or a template, often inside a condition meant for a single page type, and check any SEO app that controls robots tags.
Let AI fix it, page by page.
Paste this prompt into AI with your project open. In UpCrawler, every affected page comes with a prompt like this, already filled in with its URL.
Remove noindex from both the meta robots tag and the X-Robots-Tag header for routes that should rank. Keep it only on thin, duplicate or utility routes, and never apply it from a shared layout that covers more than intended.
After the fix
- Confirm the directive is gone with URL Inspection in Search Console, using Test live URL.
- Click Request indexing for your most important pages.
- In the Page indexing report, open URL marked 'noindex' and click Validate fix. Validation usually takes about two weeks.
Do not block the page in robots.txt at the same time. If crawling is disallowed, Google cannot fetch the page to see that the noindex is gone, and the old state can linger.
Frequently asked questions
Is "Excluded by noindex tag" an error?
Only if the page should rank. Search Console lists every noindexed URL it found, including ones that are correctly excluded, such as account pages or thank you pages. Review the list and fix only the pages you want in search. Do not use noindex to hide copies of a page you want indexed; that is a job for a canonical, as the duplicate content guide explains.
How long does it take to get back into Google after removing noindex?
It depends on how often Google visits the page: days for important pages, months for pages it rarely recrawls. Requesting indexing for key URLs and submitting an up to date sitemap speeds it up.
Does noindex also remove the page from AI Overviews?
Yes. AI features in Search only use pages that are indexed and eligible to show a snippet, so a noindexed page cannot be cited.
Share it with whoever runs your site’s SEO.
