Remove Duplicate Lines
Keep the first of each line, drop the repeats.
Lists collected from more than one place almost always contain repeats: email addresses exported from two systems, URLs gathered over several sessions, keywords merged from separate research files. Removing those repeats by hand is slow and error-prone once the list passes a few dozen lines.
This tool keeps the first occurrence of every line and discards every later copy, so the original order survives. That matters more than it seems — a deduplicated list that has also been reordered is hard to reconcile with the source it came from, and reordering is what most sort-based approaches do as a side effect.
Leading and trailing spaces are ignored when comparing, because lines that differ only by invisible whitespace are duplicates in every practical sense. An option extends that tolerance to letter case, so Apple and apple collapse into one entry when you want them to.
Examples
| What it handles | Input | Output |
|---|---|---|
| Order is preserved | a b a c b | a b c |
| Ignoring case | Apple apple Banana | Apple Banana |
| Whitespace-only differences collapse | item item other | item other |
How it works
The text is split into lines and each line is walked in order. A normalized version of the line — trimmed, and lowercased when case-insensitive matching is on — is looked up in a set of everything seen so far. If it is already there the line is dropped; otherwise it is kept and added to the set. Because lookups are constant time, the tool handles very long lists without slowing down, and because lines are emitted in the order they were first seen, the result stays aligned with your source. The counts above the output show how many lines went in, how many came out, and how many were removed.
When people use it
- Merging two exported mailing lists without sending anyone twice.
- Cleaning a keyword list collected from several research tools.
- Deduplicating URLs before a crawl or an import.
- Tidying a log or CSV column where entries repeat.
Frequently asked questions
- Which copy is kept?
- The first one. Later duplicates are removed, so the surviving line keeps its original position in the list.
- Does it sort the list?
- No, and that is deliberate. The order you pasted is the order you get back. Use the line sorter afterwards if you also want the list ordered.
- Are blank lines removed?
- All but the first. Blank lines compare as equal to each other, so repeated blanks collapse into one.
- Why does the surviving line still have its extra spaces?
- Because comparison and output are separate steps. Leading and trailing spaces are ignored when deciding whether two lines match, but the line that is kept is emitted exactly as you pasted it. So " item " followed by "item" collapses to one entry, and that entry keeps the padding of the first copy. Run the whitespace trimmer first if you want the survivors normalized as well.
- How large a list can it handle?
- Comfortably into the hundreds of thousands of lines. Everything runs locally in your browser, so the practical limit is your device rather than a server.