How to Clean Up a List by Removing Duplicate Lines
A list with duplicates in it gives the wrong answer to every question you ask of it. How many subscribers, how many unique visitors, how many distinct products - each one is inflated by however many repeats are hiding in there. The Duplicate Line Remover strips them and leaves the unique entries.
Where duplicates come from
Almost always a merge. Two exports combined, a spreadsheet pasted twice, a form submitted again because the first attempt seemed not to work, a sync that ran a second time. Nobody types the same line twice on purpose.
That is why the count is the thing that misleads. A list of four thousand email addresses with eight hundred repeats looks like four thousand subscribers, and every per-head figure calculated from it is twenty per cent wrong.
The near-duplicates that survive
Exact de-duplication removes lines that are byte for byte identical. A great many duplicates are not quite.
A trailing space. Invisible, and enough to make two identical-looking lines different. Running the list through the Whitespace Trimmer first catches most of these.
Different capitalisation. Sam@example.com and sam@example.com are the same mailbox and two different strings. Normalise with the Case Converter before de-duplicating, and the two collapse into one.
Different line endings. A file from Windows and a file from a Mac can produce lines that differ only in an invisible character at the end.
Genuinely different spellings. "Smith Ltd" and "Smith Limited" are one company and no tool will merge them for you. That is a judgement, and the only honest way to handle it is to look.
So the order matters: trim the whitespace, normalise the case, then remove duplicates. Doing it the other way round leaves work behind.
Check before you commit
Compare the before and after counts. If a list of 5,000 comes back as 4,998, that is plausible. If it comes back as 1,200, something in your data was far more repetitive than you thought - which is worth understanding rather than accepting.
And keep the original. De-duplication is destructive and there is no way to tell afterwards which lines were removed, so if you later need to know how many duplicates there were, you need the file you started with.
Common questions
Does it keep the first copy or the last?
The first, which preserves your original order. That matters if the list is in a meaningful sequence.
Will it sort my list?
No. Removing duplicates and sorting are separate operations, and keeping the order is usually what you want.
Can I find the duplicates rather than remove them?
Comparing the counts tells you how many there were. To see which lines differ between two versions, use the Text Diff Checker.
Is my list uploaded anywhere?
No. It is processed in your browser, which matters when the list is customer data.
Does it treat an empty line as a duplicate?
Several blank lines collapse to one, which is usually what you want. Removing them entirely is a whitespace job rather than a de-duplication one.
Got a list with repeats in it?