URL Deduplicator
Remove duplicate URLs from a list while keeping the first copy in its original order. Choose which URL differences to ignore; the tool compares addresses without opening them.
Treat these as the same link
What is the URL Deduplicator?
The URL Deduplicator removes repeated addresses from a list of links. Paste one URL per line, and it compares them as you type using your chosen switches. By default, it treats http and https, www and non-www, and trailing-slash variants as the same link. You can also ignore tracking tags, fragments and path letter case. The first matching link stays exactly as you entered it, and empty lines are dropped. Researchers, marketers and SEO specialists use it to tidy collected links before reviewing or sharing them. It cleans your list, not the pages behind the addresses.
Who it is for: For researchers, marketers, SEO specialists, editors, students and web developers cleaning lists of links.
How to use the URL Deduplicator
- Paste your list into “Your links, one per line”, with each address on its own line.
- Under “Treat these as the same link”, review “http and https”, “With and without www” and “With and without a slash at the end”, which start on.
- Turn on any other matching rules you need, then check “Unique links” and the count line as the result updates.
- Tap “Copy result” to copy the unique list.
Why use it
- Choose what counts as a duplicateUse six switches to decide which URL differences matter for your list.
- Keep your original linksThe first copy stays exactly as you typed it, with the original list order preserved.
- See the result as you typeThe unique list and duplicate count update as you edit your input or change a switch.
- Keep links on your deviceYour list is processed in your browser with nothing uploaded, including when you work offline after loading the tool.
At a glance
- What you put in
- URLs, one per line
- What you get
- Unique links in original order, plus unique and duplicate counts
- Limits
- Address comparison only; no page checks or query-parameter reordering
- Privacy
- Runs in your browser. Nothing you enter is uploaded or stored.
- Price
- Free for everyone. No subscriptions, no payments, no watermark.
- Works offline
- Yes, once the page has loaded
- Last checked
- by AK Shine
Worked example
https://example.com/guide http://www.example.com/guide/ https://example.com/pricing https://example.com/guide?utm_source=newsletter https://example.com/guide#tips https://example.com/pricing
https://example.com/guide https://example.com/pricing https://example.com/guide?utm_source=newsletter https://example.com/guide#tips
With the default switches: 4 unique, 2 duplicates removed; tracking tags and fragments still count as differences.
When to use something else
- Use a redirect checker or site crawler when you need to find different addresses that lead to the same page; this tool does not open links or inspect page content.
- Query-parameter order is not normalized, so ?a=1&b=2 and ?b=2&a=1 stay separate.
- For recurring jobs that need scheduled runs or command-line automation, use a script rather than a paste-and-copy workflow.
Questions people ask
How do I remove duplicate URLs from a list?
Paste one address per line into “Your links, one per line” and review the matching switches. “Unique links” updates as you type, so there is no run button to press. Tap “Copy result” when the list looks right.
Should HTTP, HTTPS, www and trailing-slash variants count as duplicates?
The switches “http and https”, “With and without www” and “With and without a slash at the end” are on by default. Turn off a switch if that difference matters for your task, because these variants do not always serve the same content.
Does removing duplicates also remove tracking parameters?
No, the retained address stays unchanged. Turn on “With and without tracking tags (utm, fbclid, gclid)” to ignore recognized tracking tags during comparison, including utm_ tags and supported click IDs. If the first copy contains tracking tags, those tags remain in the output.
Are capital letters and the part after # ignored?
Domain names are always compared without regard to capitals, but path letter case and fragments remain distinct by default. Enable “Capital and small letters in the path” or “With and without the part after #” only when those differences do not matter. They can identify different content.
Will the tool preserve the order of my links?
Yes, it keeps the first copy of each matching link exactly as you typed it and preserves the original order. Later matches are removed, and empty lines are dropped.
Is this URL deduplicator free, private and available offline?
It is free, with no subscriptions, and your links are processed in your browser without being uploaded. It also works offline once loaded.
Why do some links to the same page remain in the result?
The tool compares addresses using your selected switches, not the content behind them, so redirects can leave separate links in the result. It also does not reorder query parameters: ?a=1&b=2 and ?b=2&a=1 remain separate.
Can I extract URLs from copied text and remove duplicates at the same time?
This tool expects a prepared list with one URL per line, rather than an article or block of HTML. Extract the addresses first, then paste the list into “Your links, one per line”.