Duplicate content means the same content is available at more than one URL on your site. Google does not penalise it, but it picks one URL from each group of duplicates to index and rank, and treats the rest as copies. If it picks a different URL than you would, the wrong version shows in search, and links and signals are split across addresses you never meant to publish.
What duplicate content looks like
Most duplicates are the same page reachable through slightly different URLs:
https://example.com/shoes/trail-runner
https://example.com/shoes/trail-runner/
https://example.com/shoes/trail-runner?utm_source=newsletter
https://example.com/sale/trail-runner
https://www.example.com/shoes/trail-runner
Five addresses, one page. To a visitor they look identical. To a search engine they are five candidates for the same result.
Is there a duplicate content penalty?
No. Some duplicate content is normal on any site and does not break spam policies. Duplication only becomes a spam problem when the intent is to deceive, such as scraping other sites or creating many copies to manipulate results. Ordinary duplicates are a technical problem, not a penalty. The real costs are:
- Google chooses the URL. It picks a canonical from each group, and it can be the parameter version, the old category path or the non-HTTPS host.
- Split signals. Links to different copies are only consolidated when Google recognises them as duplicates, and you do not control when that happens.
- Wasted crawling. Every duplicate is another URL to fetch. On large sites that slows the discovery of new and updated pages.
- Confusing reports. Clicks and impressions spread across several URLs make it harder to see which pages work.
Search Console shows these cases in the Page indexing report as Duplicate without user-selected canonical and Duplicate, Google chose different canonical than user.
Why it matters for AI search
AI features in Search work from the same index as regular results. Consolidating duplicates makes it clear which URL represents the content, so the version you want is the one that can be shown and linked. Leave it to chance, and a parameter URL or an outdated path can be the address people land on.
Why duplicates happen
- URL variants:
httpandhttps,wwwand bare domain, trailing slash or not, uppercase letters. - Parameters: tracking tags, sorting, filtering and session IDs appended to otherwise identical pages.
- Several paths to one item: a product listed in multiple categories, each with its own URL.
- CMS archives: tag, category, author and date archives that list the same posts in the same order.
- Copied pages: location or product pages created from one template with only the name changed.
How to find duplicate content
- Check one page. Search Google for a distinctive sentence from the page in quotes and see which URLs appear. In Search Console, use URL Inspection and compare User-declared canonical with Google-selected canonical.
- Check every page. UpCrawler compares the content of every crawled page and flags Duplicate content when pages share identical content, listing the other URLs in each group so you can decide which one to keep.
- Check what Google grouped. In Search Console, open Indexing, then Pages, and review the two duplicate statuses above. Each listed URL shows which canonical Google picked.
How to fix duplicate content
Pick the one URL that should represent each group, then make every signal agree with it:
- Add a canonical on every duplicate pointing at the preferred URL, and a self-referencing canonical on the preferred URL itself. See missing canonical tags for how to add one in each CMS.
- Redirect what should not exist. If a duplicate has no reason to be reachable, such as an old path or the non-
wwwhost, 301 redirect it to the preferred URL. - Consolidate near copies. Merge thin pages that cover the same topic into one stronger page, then redirect the others to it.
- Link to the preferred URL in navigation, internal links and the sitemap, so your own site never points at a copy. Redirects and canonical tags are strong signals, while a sitemap entry is a weaker hint, so all of them should agree.
- Keep endless filter combinations out of the crawl. When sorting and filter parameters multiply into thousands of URLs with no search value, block those parameters in
robots.txtso crawlers spend their time on real pages. - Do not use noindex to pick a canonical. Use noindex only for pages that should not be in search at all, such as internal search results.
Build the canonical from params, never from searchParams or the incoming URL, so parameters are stripped automatically:
export async function generateMetadata({ params }: PageProps): Promise<Metadata> {
const { slug } = await params
return {
alternates: { canonical: `/shoes/${slug}` }
}
}
Next.js already redirects /about/ to /about by default. trailingSlash: true in next.config.ts flips the direction, and skipTrailingSlashRedirect turns the redirect off. Pick one form and keep every internal <Link href> in it, so no link starts with a redirect.
- Your SEO plugin outputs canonicals on posts and pages. Check that archive pages you do not need in search, such as date archives, are set to noindex in the plugin settings.
- Under Settings, then General, make sure WordPress Address and Site Address use the same final host, including
httpsandwww. - Media attachment pages create a thin duplicate for every uploaded file. Yoast SEO and Rank Math can redirect them to the file itself.
Shopify serves every product at /products/handle and also at /collections/collection/products/handle when it is linked from a collection. The theme's canonical already points at /products/handle, but internal links still lead crawlers to the longer URLs. In the product card snippet, remove the within filter so links go straight to the canonical:
<a href="{{ product.url }}">{{ product.title }}</a>
Instead of {{ product.url | within: collection }}.
Let AI fix it, page by page.
Paste this prompt into AI with your project open. In UpCrawler, every affected page comes with a prompt like this, already filled in with its URL.
Pick the preferred URL per duplicate set and canonical the rest to it. Where the pages have no reason to exist separately, consolidate the content onto the preferred URL, then delete the others and repoint every internal link at the survivor. Never add a redirect of any kind.
Frequently asked questions
Does duplicate content hurt SEO?
It does not cause a penalty, but it lets Google choose which URL ranks, splits signals between copies and wastes crawling. Canonical tags and consistent internal links fix most cases.
What does "Duplicate, Google chose different canonical than user" mean?
You declared a canonical, but Google decided another URL in the group is the better representative. It usually means your other signals disagree: internal links, the sitemap or redirects point at a different version, or the pages are not as identical as you think.
Should I use a canonical tag or a 301 redirect?
Use a 301 when the duplicate should not be reachable at all. Use a canonical when both URLs need to keep working, such as a filtered view or a page shared with tracking parameters.
Is content copied from my site by other websites a problem?
Usually not, because search engines try to show the original source. If a scraper outranks you with your own content, you can request removal of the copy through Google's legal removal process.
Share it with whoever runs your site’s SEO.
