How to Remove Duplicate Lines From a List
Duplicate lines appear whenever a list is assembled from several sources: merged spreadsheets, exported keyword reports, copied log output, or survey responses. Removing them by hand is slow and error-prone. Here are the three reliable methods, from fastest to most manual.
Method 1: Use an online dedupe tool (fastest)
- Open the Remove Duplicate Lines tool.
- Paste your list, one item per line, into the left box.
- Choose your options. Ignore case treats "Apple" and "apple" as the same line. Trim whitespace ignores accidental leading and trailing spaces. Remove empty lines strips blank rows.
- The clean list appears instantly on the right. Copy it, or download it as a .txt file.
By default the tool keeps the first occurrence of each line and drops later copies, preserving your original order. If a later entry should win, switch the keep mode to last occurrence.
Method 2: Excel or Google Sheets
Paste the list into column A, select it, then use Data → Remove Duplicates (Excel) or Data → Data cleanup → Remove duplicates (Google Sheets). This works well for small lists, but it changes your spreadsheet layout and requires a file round-trip.
Method 3: Command line
On Mac or Linux, sort -u list.txt removes duplicates but also sorts the list. To preserve order, use awk '!seen[$0]++' list.txt. Both require a terminal and a saved file.
Common pitfalls
- Case differences: "SEO" and "seo" are different lines unless you enable ignore-case.
- Invisible spaces: "keyword" and "keyword " look identical but differ. Enable trim-whitespace.
- Blank lines: empty rows count as lines. Remove them before or during deduping.
- Sorting: sorting makes duplicates easy to spot, but destroys your original order. Only sort when order does not matter.
Ready to clean your list? Try the free Remove Duplicate Lines tool →