Remove duplicate lines
Paste a list with duplicates and get back a clean list with each entry kept only once. Order is preserved by default; sorting and case-insensitive matching are optional.
What this tool does
Paste a list of items (one per line) and the tool returns the same list with every duplicate entry removed. By default, the first occurrence of each item is kept and the others are dropped; the rest of the list retains its original order.
Options
- Trim whitespace. Treats
"alpha"and" alpha "as the same item. On by default. - Case-insensitive. Treats
"Alpha"and"alpha"as the same item. The first-encountered casing is preserved in the output. - Drop empty lines. Removes blank lines from the result. On by default.
- Sort alphabetically. Sorts the de-duplicated list in ASCII order. Off by default; turn this off if line order matters.
When to use it
Anytime you have a list with redundant entries. Concrete examples:
- Merging two email lists before sending a campaign.
- Cleaning up a list of URLs after scraping.
- De-duplicating log lines to focus on unique errors.
- Compiling a vocabulary list from multiple sources.
- Preparing a flat list for import into a spreadsheet or database that disallows duplicates.
Performance
The tool uses a JavaScript Set for membership checks, which is constant-time on average. Lists with millions of lines de-duplicate in well under a second on a modern laptop. The bottleneck is your browser's rendering of the result textarea, not the de-duplication itself.
What it doesn't do
This is a strict, exact-match de-duplicator. It doesn't perform fuzzy matching, regex substitution, or column-aware de-duplication. If your data is in a CSV and you want to de-duplicate by a specific column, paste only that column, de-duplicate it, then re-join — or use a spreadsheet's built-in de-duplication feature.
Privacy
The list is processed in your browser. The list never leaves your device.