A robots.txt error is a rule in your /robots.txt file that blocks crawlers from pages you want in search, or a file so broken that crawlers cannot follow it. One leftover Disallow: / from a staging site is enough to stop Google crawling everything, and Search Console then reports pages as Blocked by robots.txt. The fix is to correct the rule at its source and confirm it in the Search Console robots.txt report.
What robots.txt does
robots.txt is a plain text file at the root of each host, such as https://example.com/robots.txt. It tells crawlers which paths they may fetch. Each group starts with a User-agent line, followed by Allow and Disallow rules:
User-agent: *
Disallow: /cart
Disallow: /search
Sitemap: https://example.com/sitemap.xml
This is the line that hides a whole site, often copied from staging during a launch:
User-agent: *
Disallow: /
robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it, usually with no description. That is the Search Console status Indexed, though blocked by robots.txt.
Why it matters
- Blocked pages cannot rank on their content. Google cannot read a page it is not allowed to fetch, so the page either drops out or shows up as a bare URL.
- Blocked CSS and JavaScript break rendering. Google renders pages like a browser. If the files that build the layout are disallowed, it may see an empty or broken page.
- A blocked page hides its noindex. Google has to fetch a page to see a noindex tag. Blocking it in robots.txt as well means the noindex is never read.
- AI search reads the same file. Google's AI features rely on the regular Googlebot crawl, and AI search crawlers such as OAI-SearchBot and PerplexityBot follow robots.txt too. A rule that blocks them keeps your pages out of the answers they build.
- A failing file pauses crawling. If
/robots.txtreturns a server error, Google first treats the whole site as disallowed and then falls back to the last copy it fetched. A 404 is safe: it means there are no restrictions.
Why it happens
- Staging rules shipped to production. The most common and most costly mistake.
- Over broad paths. Rules match by prefix, so
Disallow: /palso blocks/productsand/pricing. - Wrong case or path. Paths are case sensitive.
Disallow: /Admindoes not block/admin. - Blocking asset folders such as
/wp-includes/or/assets/, which are needed for rendering. - Using robots.txt to deindex. Google stopped supporting
noindexinside robots.txt in 2019, so those lines do nothing. - Conflicting groups. A crawler follows only the most specific
User-agentgroup that matches it, so rules in the*group do not apply to Googlebot once aGooglebotgroup exists. - Typos and stray characters, such as a missing colon or a rule placed before any
User-agentline.
How to find robots.txt errors
- Open the file. Visit
/robots.txton every host you serve, includingwwwand any subdomain, because each one has its own file. - Check the Search Console robots.txt report. Under Settings, the robots.txt report shows the file Google last fetched, when it fetched it and any lines it could not parse.
- Inspect a URL. The URL Inspection tool tells you whether a specific page is blocked and which rule blocks it.
- Check the Page indexing report for Blocked by robots.txt and Indexed, though blocked by robots.txt, then compare the URLs with the pages you want ranked.
- Check it in UpCrawler. When you add a site, UpCrawler reads your robots.txt and shows Restrictions found when its rules block crawlers, so you can catch a stray
Disallowbefore the first crawl.
How to fix it
- Remove any rule that blocks pages you want ranked, starting with
Disallow: /. - Unblock CSS, JavaScript and image folders that pages need to render.
- Narrow prefix rules. End a folder rule with a slash, and use
$to match the end of a URL, for exampleDisallow: /*.pdf$. - Use noindex, not robots.txt, to keep a page out of search, and leave that page crawlable so the tag can be read.
- Keep one group per crawler and add a
Sitemap:line with the absolute URL of your sitemap. - Recheck the report. Ask Google to recrawl the file from the robots.txt report so the fix takes effect sooner.
Generate the file from app/robots.ts and allow production only, so a staging rule can never ship:
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
return process.env.VERCEL_ENV === 'production'
? {
rules: { userAgent: '*', disallow: ['/cart', '/search'] },
sitemap: 'https://example.com/sitemap.xml'
}
: { rules: { userAgent: '*', disallow: '/' } }
}
Replace VERCEL_ENV with the variable your host sets. Delete any static public/robots.txt, which conflicts with the generated route.
WordPress serves a virtual robots.txt when no file exists. Edit it through your SEO plugin: in Yoast SEO, Tools, then File editor; in Rank Math, General Settings, then Edit robots.txt. A sensible default:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xml
If a physical robots.txt file exists in the site root, the server sends it instead and the virtual rules are ignored. Edit or delete that file. Also check Settings, then Reading, in case "Discourage search engines from indexing this site" is still ticked from development.
Shopify generates robots.txt with sensible defaults. To change it, add a robots.txt.liquid template in the theme code editor and edit the default groups instead of replacing them:
{% for group in robots.default_groups %}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.user_agent.value == '*' -%}
{{ 'Disallow: /*?q=*' }}
{%- endif -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}
If pages were blocked after a theme change, delete the custom template to go back to Shopify's defaults.
Let AI fix it, page by page.
Paste this prompt into AI with your project open. In UpCrawler, every affected page comes with a prompt like this, already filled in with its URL.
Correct the invalid directives and stray characters, keep one group per user-agent, and confirm nothing you want indexed is disallowed. Never ship Disallow: / to production.
Frequently asked questions
What does "Blocked by robots.txt" mean in Search Console?
Googlebot found the URL but a rule in your robots.txt stopped it from crawling the page. If the page should rank, remove or narrow the rule. If it should stay out of search, unblock it and add a noindex tag instead.
What does "Indexed, though blocked by robots.txt" mean?
Google indexed the URL from links pointing at it without being able to read the page. To remove it, allow crawling and add a noindex tag. To keep it, remove the blocking rule so Google can read the content.
How do I test my robots.txt?
Open the robots.txt report in Search Console to see the version Google fetched and any parse errors, then use the URL Inspection tool on the pages you care about to confirm they are allowed.
Can I block AI crawlers in robots.txt?
Yes. Add a group for the crawler's user agent with Disallow: /. Check what each bot is for first: OpenAI's GPTBot collects training data, while OAI-SearchBot powers ChatGPT search. Blocking a search crawler keeps your pages out of its answers and citations.
Share it with whoever runs your site’s SEO.