Remove duplicate lines online

How to remove duplicate lines

1. Paste your list, one item per line, into the box.
2. Decide whether duplicates should be matched case-insensitively (recommended for most lists).
3. Decide whether to trim whitespace before comparing lines.
4. Click Remove duplicates.
5. Check the summary showing how many lines were kept and how many duplicates were removed.

The first occurrence of each unique line is kept and later duplicates are removed; the original order of the remaining lines is preserved.

How duplicate detection works

Each line is compared against the lines seen so far. If case-insensitive matching is on, "Apple" and "apple" are treated as the same line. If whitespace trimming is on, "apple" and "apple " (with a trailing space) are treated as the same line. The first time a line appears it is kept; every later occurrence of that same line is removed. Lines that are unique to begin with are always kept, in their original order.

Best practices

• Keep both options on for most lists (email addresses, tags, names) since inconsistent capitalization and stray spaces are extremely common in pasted data.
• Turn case-insensitive matching off if case is meaningful in your data, such as a list of exact code identifiers or file paths where case changes the meaning.
• Sort the list first with sort text lines if you want the deduplicated result in alphabetical order rather than original order.

Common mistakes to avoid

• Assuming this also sorts your list. It only removes duplicates and preserves original order; use the sort tool separately if you also want alphabetical order.
• Turning off whitespace trimming without checking your data first, which can leave near-identical lines (one with a trailing space, one without) both in the result.
• Not checking the removed-count summary, which is a quick way to sanity-check whether the result matches what you expected.

Real-world use cases

• Cleaning an email or contact list exported from a spreadsheet with accidental duplicate entries.
• Deduplicating a tag or keyword list before importing it elsewhere.
• Merging two lists (paste both, one after another) and removing the overlap.
• Cleaning up a URL or log-entry list before further processing.

Limitations

This tool compares whole lines only; it cannot detect near-duplicates that differ by more than case and surrounding whitespace (for example, "John Smith" and "J. Smith" are treated as different lines). It also does not sort the result — order is preserved as-is except for removed duplicates.

Privacy

Text you paste is processed locally in your browser. It is not sent to a server, logged, or stored. See the privacy policy for the site's full data handling approach.

Troubleshooting

• If lines that look identical aren't being merged, check for invisible trailing whitespace or turn on whitespace trimming.
• If lines with different capitalization aren't being merged, turn on case-insensitive matching.
• If you need the result sorted too, run it through sort text lines next.

Duplicate lines vs. duplicate records

This tool compares plain text lines, so it's a good fit for simple lists like email addresses, tags, or URLs where each line is one complete value. For spreadsheet-style data with multiple columns (like "name, email, phone" per line), deduplicating by full line will only catch rows that are identical across every column — two rows with the same email but a different phone number would both be kept, since the lines aren't identical. If you need to check the shape of that kind of data first, preview it with the CSV viewer before deciding how to deduplicate it.

Conclusion

Use this tool any time you have a pasted list with accidental duplicates. Combine it with sort text lines if you also want the result in alphabetical order.