Remove Duplicate Lines
The first time a line appears it stays. Every repeat after that goes. The order never changes.
Processed in your browser. No signup.
What this tool does
Paste a list and every line that has already appeared is dropped. The first occurrence of each line is kept exactly as you wrote it, in its original position, so the surviving list reads in the same order as the one you pasted.
How to use it
- Paste your list, one item per line.
- Read the counts: lines in, unique lines, duplicates removed.
- Adjust the options if items differ only by case or by stray spaces.
- Copy the result.
Example
The list apple, banana, apple, pear, banana becomes apple, banana, pear — 5 lines in, 3 unique, 2 duplicates removed. Note that banana keeps its original second position rather than moving.
Options
Case sensitive
Off by default, so Apple and apple count as the same line. Turn it on when case carries meaning, such as in identifiers or passwords.
Ignore spaces at the ends of lines
On by default. Lines are compared after trimming, which catches duplicates that differ only by an invisible trailing space — but the line that is kept still contains its original spacing. If you also want the spacing gone, run the result throughText Cleaner.
Remove blank lines
Off by default, so blank lines are left exactly where they are and never treated as duplicates of each other. Turn it on to collapse the list into a solid block.
Order and sorting
Nothing is sorted here. Deduplicating and sorting are separate decisions, and doing both at once makes it hard to see what actually changed. If you want the result alphabetised, send it to Sort Lines afterwards.
Useful cases
- Cleaning a keyword list exported from a research tool.
- Removing repeated email addresses or URLs before an import.
- Tidying a log or an export where the same row appears several times.
- Finding out how many genuinely distinct values a list contains.
Large lists
Comparison uses a hash set, so the work grows in proportion to the number of lines rather than the square of it. Lists of several hundred thousand lines are handled without trouble.
Privacy
Your list is processed in your browser and never uploaded.