Match sensitivity
Loose match catches reworded/paraphrased duplicates but can over-flag genuinely distinct pages on a similar topic. Strict match only catches near-identical text. Balanced is the right default for most audits.
Sitemap Duplicate Content Scanner
Paste a sitemap URL and get every near-duplicate and cannibalizing page pair on your site — compared page by page, right in your browser.
Sitemap Duplicate Content Scanner
Paste a sitemap URL. It reads every page, compares the actual body content of each pair, and flags pages that are near-duplicates of each other — plus pages that may be competing for the same query.
Reading the sitemap…
Still working — larger sitemaps take longer to compare.
Note: comparisons run on stripped body text only (nav, header, footer, scripts removed), so shared templates don't count as duplicates. Sitemap and page fetching happens directly from your browser first; if a site blocks that (CORS), it's retried through our fetch proxy automatically — pages still blocked after both attempts show under "Couldn't fetch" rather than being silently skipped. "Same title/H1" is a lightweight signal from on-page text, not real ranking data — treat it as a lead to check in Search Console, not a verdict. Comparison itself always happens locally in your browser; only the raw page fetch is proxied when needed.
processed in your browser · never uploaded
Scope
What it is, and what it isn't.
Every tool on ToolsVale is scoped to one job. Here is exactly what this one does, and what it does not — so you know whether it fits before you start.
What it is
- What is a duplicate content scanner? A duplicate content scanner reads through every URL in your sitemap, pulls the actual page content, and compares pages against each other to find ones that are too similar — whether that's accidental (templated product pages, copy-pasted service pages) or a sign two pages are competing for the same search query. This tool does that comparison locally using shingled MinHash signatures, so it can check hundreds of page pairs without sending your content to a server.
What it isn't
- What this tool doesn't do It isn't a plagiarism checker — it won't tell you if another site copied your content (for that, use a tool built for external web-wide comparison). It also isn't a ranking or Search Console replacement: the cannibalization flag is a lightweight signal based on matching titles/H1s, meant to point you toward pages worth checking manually, not a verdict on how Google is actually treating them.
Comments
What people made of this one.
Notes from other visitors. Sign in with Google to add yours, or send a private suggestion straight to the developer.
Sign in to leave a comment. We only use Google — no passwords, no email spam.
Sign in with GoogleNo comments yet. Be the first.
How this compares
This tool vs. the alternatives.
A same-page side-by-side against the tools a visitor would otherwise pick. Every value is verifiable on the competitor's current site — flag anything that looks stale.
| Feature | Sitemap Duplicate Content Scanner this tool | Copyscape | Siteliner | Screaming Frog SEO Spider |
|---|---|---|---|---|
| Compares | Your own site's pages against each other | Your content against the wider web | Your own site's pages against each other | Crawl data, incl. near-duplicate detection |
| Price | Free | Paid (per search / subscription) | Free tier + paid | Free up to 500 URLs, then paid |
| Setup | None — paste sitemap URL | Account required | None | Desktop app install |
| Runs where | Your browser | Their servers | Their servers | Your desktop |
Compared on public plans as of the last review date. Competitor features change — if you spot a stale row, please flag it and we'll re-verify.
How to use it
Sitemap Duplicate Content Scanner
Sitemap Duplicate Content Scanner
in three moves.
01
Find your sitemap URL
Usually yoursite.com/sitemap.xml or sitemap_index.xml. Check your site's robots.txt if you're not sure — it's listed there.
02
Pick a page limit and sensitivity
Start with 200 pages and "Balanced match." Use "Loose match" if you suspect reworded (not copy-pasted) duplicates, or "Strict" to surface only near-identical pages.
03
Run the scan
The scanner reads your sitemap (including nested sitemap indexes), fetches each page, strips out nav/header/footer/scripts, and compares what's left.
04
Review and export
Filter by High overlap, Moderate, Cannibalization, or Couldn't fetch. Download a CSV or copy the flagged URLs straight into your task tracker.
Formats & limits
Format field guide
What goes in, what comes out.
| Input | Output | Typical saving | Best for |
|---|---|---|---|
| Any public sitemap.xml or sitemap index URL | Table, CSV, or plain-text flagged URL list | 50–300 pages in under a couple of minutes | Sites doing a content audit, post-migration cleanup, or checking for keyword cannibalization before a content refresh |
Why this one
Built against the ways these tools disappoint.
We ran the same files through the popular alternatives first. These are the gaps we found, and what this tool does instead.
01
Most free duplicate-content checkers compare only meta titles/descriptions, or require you to paste in page text one at a time.
We crawl the whole sitemap automatically and diff actual rendered body content, page against page.
test: Run it against your own sitemap.xml right now — no account, no page limit lower than 300.
02
Server-side scanners send your page content to a third-party backend for comparison.
The similarity comparison itself runs locally in your browser (Web Worker); only the raw HTML fetch is proxied when a site's CORS policy blocks a direct read, and that proxy doesn't log or store anything.
test: Open dev tools during a scan — the compare step never issues a network request.
03
Flagging "duplicate" pages without distinguishing shared templates from actually duplicated content.
Nav, header, footer, and scripts are stripped before comparison, so two pages sharing a template don't falsely trigger a match.
test: Check the "Signal" column — matches include a same title/H1 flag only when that's independently true, not because of template overlap.
Key Factors
What affects your results
Sites that block automated fetching
Some sites block cross-origin or bot-like requests entirely. We retry through a fetch proxy automatically, but a small number of pages may still show under "Couldn't fetch" — that's the site's server config, not a scanner limitation.
Thin or short pages
Pages with very little body text (under ~40 characters after stripping) aren't compared, since there isn't enough content to produce a meaningful similarity score.
The detail
How this actually works.
Find Duplicate Content on Your Site — Free Sitemap Scanner
If you’ve been publishing for a while — product pages, location pages, blog posts, service pages — odds are some of them have started to look a lot alike to a search engine, even if they don’t feel that way to you. A duplicate content checker that scans your whole sitemap is the fastest way to find out where that’s happening, without manually opening dozens of tabs to compare pages by eye.
This tool does exactly that. Paste in your sitemap.xml, and it reads every page listed, strips out the navigation and boilerplate, and compares what’s left — the actual body content — page against page. Anything that comes back too similar gets flagged, along with pages that may be quietly competing against each other for the same search query.
How the Duplicate Content Scanner Works
- Reads your sitemap. Works with a single sitemap or a sitemap index that links out to several child sitemaps — it follows those automatically.
- Fetches each page. Pages are fetched directly from your browser first. If a site blocks that (CORS), the fetch is retried through a proxy automatically, so pages still get checked either way.
- Strips the template. Navigation, headers, footers, and scripts are removed before comparison, so two pages sharing the same layout don’t get falsely flagged just for using the same theme.
- Compares content locally. The actual similarity scoring runs inside your browser using a Web Worker — your page content is never sent off to a server for the comparison step itself.
- Flags what matters. Results are sorted into high overlap, moderate overlap, and possible cannibalization (pages sharing the same title or H1), so you know where to look first.
Why Duplicate Content Is Worth Checking
Duplicate or near-duplicate pages create a few specific problems for SEO:
- Search engines have to choose which version to show, and it might not be the one you’d pick.
- Link equity gets split across multiple similar pages instead of consolidating behind one strong page.
- Keyword cannibalization happens when two pages target the same query, so they end up competing with each other instead of ranking together.
- Thin, templated pages — think near-identical product or location pages — can drag down how a search engine evaluates your site as a whole.
None of this requires malicious copy-pasting. It usually creeps in gradually: a template reused a few too many times, an old page that was never properly consolidated after a rewrite, or two blog posts written months apart that ended up covering the same ground.
What You’ll See in Your Results
Each scan gives you:
- A similarity percentage for every flagged page pair
- A same title/H1 flag where two pages might be cannibalizing the same keyword
- A list of pages that couldn’t be fetched, so nothing gets silently skipped
- A one-click CSV export or a copyable list of flagged URLs to hand off to your content team
How to Use the Sitemap Duplicate Content Scanner
Paste your sitemap URL — usually something like yoursite.com/sitemap.xml or sitemap_index.xml — into the box above. Choose how many pages to compare (up to 300) and how strict the match should be:
- Loose match — catches reworded or paraphrased duplicates, useful if you suspect content was spun rather than copy-pasted
- Balanced match — the right default for most audits
- Strict match — flags only near-identical pages
Hit scan, and results populate directly on the page — no email required, no signup.
Frequently Asked Questions
Does this send my content to a server?
No. The similarity comparison itself runs in your browser. Only the raw page fetch is proxied when a site blocks direct cross-origin requests, and that proxy doesn’t log or store anything — the URL and response pass straight through and are discarded.
What counts as duplicate vs. moderate overlap?
Page pairs scoring 60% similarity or higher are flagged as high overlap. Anything at or above your chosen sensitivity threshold but below that is moderate overlap.
Why do some pages show “Couldn’t fetch”?
Either the target site blocked both the direct and proxied fetch attempts, the request timed out, or the page didn’t have enough text to compare. These are listed separately rather than silently dropped from your results.
Is this the same as a plagiarism checker?
No. Plagiarism checkers look for your content copied elsewhere on the web. This tool checks for duplication within your own site — pages that overlap with or cannibalize each other.
Is there a page limit?
You can scan up to 300 pages per run. For larger sites, run separate scans against individual sub-sitemaps (product sitemap, blog sitemap, etc.) rather than one massive combined one.
Ready to check your own site? Paste your sitemap URL above and run your first scan — it’s free, and nothing you fetch leaves your browser for comparison.
Last reviewed August 2026 · this tool runs on-device.
FAQ
The questions people actually type in.
Does this tool send my page content to a server?
No. Comparison runs entirely in your browser using a Web Worker. Only the raw page fetch is proxied when a site blocks direct cross-origin requests, and that proxy doesn't log or store the URL or response.
What counts as "duplicate" vs "moderate overlap"?
Similarity scores of 60% or higher are flagged as high overlap; anything at or above your chosen sensitivity threshold but below 60% is moderate. You can adjust the threshold (loose/balanced/strict) before scanning.
Why do some pages show "Couldn't fetch"?
Either the site blocked both the direct browser fetch and the proxy fallback, the request timed out, or the page had too little text to compare. It doesn't mean those pages are fine — just that we couldn't evaluate them.
How is this different from a plagiarism checker like Copyscape?
Copyscape checks whether your content has been copied by other sites across the web. This tool checks for duplication within your own site — pages competing with each other or templated content that reads as near-identical to search engines.
What does "possible cannibalization" mean?
It flags page pairs that share the same