📝 Text Deduplication Tool
How It Works
Paste into the left panel, pick a delimiter (newline, comma, semicolon, whitespace, or any custom string), and hit Deduplicate (Ctrl+Enter works too). The right panel shows the unique list with an "original → deduplicated" line count. Copy it or download as .txt — always newline-separated for easy downstream processing. Everything runs in your browser (up to 200,000 characters); nothing is uploaded.
The Same Operation in Other Tools
Deduplication is a universal operation — knowing the equivalents helps you pick the right tool for the job:
| Tool | How | Order kept? |
|---|---|---|
| This page | paste → Deduplicate | Yes (first seen) |
| Excel | Data → Remove Duplicates | Yes |
| SQL | SELECT DISTINCT col FROM t | No (unless ORDER BY) |
| Python | list(dict.fromkeys(items)) | Yes; plain set() doesn't |
| Unix shell | sort file | uniq | No — uniq alone only removes adjacent duplicates |
The classic gotcha in that table: uniq without sort misses non-adjacent duplicates — a mistake this tool can't make, since it hashes every line.
What "Duplicate" Means Here
- Exact match after optional case-folding and trimming — "New York" and "new york " collapse only if both options are on. Fuzzy matching (typos, synonyms) is out of scope by design.
- First occurrence wins: the surviving entry keeps its original casing and its original position, which preserves priority order in sign-up lists.
- Sort is optional and orders by Unicode code point — fine for ASCII, but note that accented letters sort after "z" (É > z), so locale-aware sorting belongs in a spreadsheet.
- Invisible whitespace — trailing spaces, tabs, and full-width spaces (U+3000) pasted from spreadsheets or chat apps — is the top cause of "duplicates that won't dedupe." Turn on Trim Spaces.
Frequently Asked Questions
How large a list can it handle?
Up to 200,000 characters — typically thousands to tens of thousands of lines — with auto-truncation beyond. For millions of records, use SQL DISTINCT or Python; this tool is built for quick, no-setup cleanups.
Can the custom delimiter contain regex characters?
Yes — the delimiter is treated literally (special characters are escaped automatically), so | or . split exactly on those characters.
Why does the download always use newlines?
One-item-per-line is the most tool-friendly format — it feeds directly into spreadsheets, scripts, and diff tools regardless of what delimiter the input used.
Is my pasted data private?
Yes — deduplication runs entirely in your browser. Nothing is transmitted to a server, which is also why it's fast.