A repeated product ID is easy to delete. A list with repeats scattered between useful entries needs more care, especially if its order matters. When I remove duplicate lines from text, I want to keep the first copy without changing the spelling or moving anything around. I built Remove Duplicate Lines for that kind of cleanup, and I test my tools on a budget Android phone and a laptop. Here’s how I approach the job, including the checks I make before trusting the result.
How do I remove duplicate lines from text online?
I use Remove Duplicate Lines for a plain list with one entry per line. That might be keywords, product IDs, reading references or copied notes. My first decision isn't which button to press. It's what I want a duplicate to mean: an exact repeated line, or a match after changing spaces or capitalization.
My step-by-step cleanup
- Save an unchanged copy of your original text before editing.
- Open Remove Duplicate Lines and paste your list into its text input.
- Keep each complete item on its own line.
- Check whether blank lines are meaningful separators or unwanted gaps.
- Run the duplicate removal and review the resulting text.
- Check that the first occurrence stays where you expect.
- Look for unexpected changes to case, spacing and blank lines.
- Copy the cleaned result into a separate document, then compare it with your original before replacing anything.
I tried this workflow with Remove Duplicate Lines. My advice is to test a small sample before pasting an important list. Include an exact repeat, a capitalization difference and a blank line. That lets you judge the behavior you need instead of assuming every cleanup tool handles those details the same way.
Tip from me: I keep the original open beside the result on my laptop. On my phone, I save the original separately before copying the cleaned text, so my clipboard isn't my only backup.
What counts as a duplicate line?
For exact matching, I treat the whole line as the value. Matching words inside different sentences don't make those sentences duplicates. Likewise, a product name followed by a different product ID is a different line. The cleanup should filter repeated entries, not guess which details you meant to ignore.
| Line comparison | Exact duplicate? | My decision |
|---|---|---|
| apple compared with apple | Yes | Keep one copy if repetition has no meaning. |
| Apple compared with apple | No | Decide whether capitalization matters. |
| apple compared with apple followed by a space | No | Decide whether to trim surrounding spaces. |
| apple compared with apple, | No | Keep punctuation unless I deliberately change it. |
| A blank line compared with another blank line | Potentially | Check whether document spacing must remain. |
Exact matching versus changing the text first
I separate deduplication from normalization, which means making text follow the same format. Changing Apple to apple is normalization. Removing the later apple is deduplication. Doing both can be useful, but they are different edits. I want you to know which edit caused something to disappear.
If case genuinely doesn't matter, I can prepare a copy with Case Converter before removing repeats. I wouldn't do that casually with identifiers or other text where capitalization carries meaning. My preference is to make the smallest change that solves the problem, then check the result before doing another cleanup.
How do I keep the first copy, last copy or only unique values?
The word unique causes confusion because it can describe different results. I might want one copy of every value. Or I might want only values that never repeat. Those aren't interchangeable. Before choosing a method, I write down the result I expect from a short sample.
| Goal | Expected output order | What I keep in mind |
|---|---|---|
| Keep the first occurrence | apple, banana, orange | Preserve the first appearance of each value. |
| Keep the last occurrence | apple, orange, banana | Preserve the positions of the last appearances. |
| Keep only values occurring once | orange | Remove every copy of repeated values. |
| Return unique values in sorted order | apple, banana, orange | Sorting changes order when the input isn't already sorted. |
| Collapse adjacent repeats only | apple, banana, apple, orange, banana | This sample has no adjacent repeats. |
Keeping the last occurrence needs an explicit method
If I need the last occurrence, I don't assume a keep-first operation can do it directly. One approach is to reverse the line order, keep the first occurrence in that reversed list, then reverse the result again. I test that sequence on a sample because forgetting the final reversal leaves a different ordering.
For values that occur exactly once, Google Sheets offers the exactly_once argument in UNIQUE. I use that distinction carefully: it removes both copies of a repeated value rather than saving one. If you're cleaning a list for reuse, ordinary keep-first deduplication is usually the closer match. If you're investigating exceptions, exactly-once filtering may be what you meant.
Why do identical-looking lines remain after cleanup?
When a visible repeat survives, I first check whether the lines really contain the same characters. A trailing space is hard to spot. A tab can resemble several spaces. And text copied from a website or PDF can include invisible characters that an exact comparison correctly treats as different.
The checks I make before trying again
- I compare the beginning and end of each suspicious line for spaces.
- I check whether tabs and spaces have been mixed.
- I compare capitalization and punctuation rather than reading only the words.
- I inspect line boundaries, including the final line, before assuming the comparison failed.
- I consider non-breaking spaces and other invisible Unicode characters if ordinary trimming doesn't explain it.
Google specifically says its Sheets Trim whitespace command doesn't remove non-breaking spaces. I keep that limitation in mind when text still looks duplicated after trimming. You can read the details in Google’s whitespace and duplicate-removal help. A cleanup command finishing successfully doesn't prove every invisible difference is gone.
I may use Text Cleaner or Find and Replace for a deliberate preparation step. But I work on a copy and inspect the changes. Removing all spaces would turn a spacing problem into damaged text. For accented or non-English content, I also test representative lines instead of judging the whole file from simple English examples.
How do I remove duplicate lines from a text file safely?
For an ordinary text file, I can open a copy in a text editor, copy its contents into Remove Duplicate Lines, then save the reviewed result under a new filename. I don't promise a browser can handle every file size. For large files, available device memory and the chosen method matter, so I keep a local option ready.
Keep order with awk, or sort deliberately
If your system has awk, a common order-preserving command is awk '!seen[$0]++' input.txt > output.txt. It prints the first occurrence of each exact line. I use different input and output filenames, then inspect output.txt before deciding whether it should replace the original.
If I want sorted unique lines instead, GNU sort -u input.txt > output.txt is the relevant choice. It sorts as well as removing repeats. GNU uniq input.txt > output.txt does something narrower: it collapses adjacent repeated lines. It won't remove separated copies just because they appear somewhere in the same file.
- I use awk when keeping the first occurrence and original order is the goal.
- I use sort -u when I actually want sorted output, while checking locale-sensitive text.
- I use uniq when only consecutive repeats should collapse.
Tip from me: I never redirect output into the same file I'm reading. A command such as sort -u input.txt > input.txt can destroy the input. A separate output file makes review possible.
How do I remove duplicate rows in Excel or Google Sheets?
I choose a spreadsheet method when each entry has related columns, such as an ID beside a name and a note. That relationship matters. A plain-text line remover doesn't decide which columns form a record, and copying only one column out for cleanup can separate it from the information beside it.
Excel comparison columns can remove a whole row
In Excel, I copy the original data first, select the complete table, then use Remove Duplicates. If I want matches based on one column, I select that column as the comparison key in the dialog. Excel keeps the first occurrence, but a matching record is removed as a whole row within the selected data, including its other columns.
That means an identical ID with a different note can still lead to a deleted record when only the ID is checked. I review those differences before removal. Microsoft recommends copying the original because duplicate deletion is permanent. I don't treat the surviving row as proof that every discarded detail was unnecessary.
Sheets and Power Query don't compare everything alike
Google Sheets UNIQUE returns rows in their first-appearance order. Its Remove duplicates command treats identical values with different capitalization, formatting or formulas as duplicates. Power Query differs: its comparison is case-sensitive, and Microsoft doesn't guarantee which duplicate it keeps. I check the exact feature, not just the application name, before relying on case handling or a particular surviving record.
What mistakes should I avoid when deleting repeated lines?
My biggest caution is that repetition isn't always an error. Repeated log entries may describe separate events. Repeated survey answers may be valid responses. A repeated phrase in notes may show frequency. I keep the source untouched whenever those repeats might carry information that a cleaned list would hide.
- Don't sort just to remove duplicates if the current order represents priority or sequence.
- Don't remove line breaks first. That can join separate entries into one line before comparison.
- Don't assume a wrapped display line is a separate record. Check the actual line boundaries.
- Don't use plain-text deduplication as a substitute for understanding CSV records.
- Don't normalize case, punctuation or spaces unless those differences are irrelevant to your task.
CSV needs particular care because quoted fields can contain embedded newlines. A visible line isn't necessarily a complete row. Headers and commas also affect how the data should be interpreted. I use a record-aware workflow for structured data instead of assuming that deleting repeated physical lines is equivalent to removing duplicate records.
For URL lists, I separate repeated text from address cleanup. Identical URLs are straightforward line matches, but different addresses need closer judgment. I use URL Deduplicator when the task is about URL entries, and I don't assume similar-looking addresses are interchangeable. Likewise, I use Sort Lines Alphabetically only after deciding that alphabetical order is useful.
How do I check the result and protect sensitive text?
I don't judge a cleanup only by whether the output looks shorter. I compare the beginning, middle and end with the original, then inspect known repeats and near-matches. If preserving order was the goal, I check that the surviving entries follow their original first appearances rather than an alphabetical sequence.
- Confirm that each surviving item is complete and hasn't been joined to its neighbour.
- Check blank lines, capitalization and surrounding spaces against your chosen rules.
- Inspect known duplicates and a few entries that should remain distinct.
- Save the result separately and reopen it in the application where you will use it.
For sensitive text, I check the specific tool's processing and privacy information before pasting. A statement about one website processing locally isn't evidence about another website. I won't make a blanket promise that online cleanup is private, that nothing is stored, or that every file size is supported.
If the text includes passwords, API keys or restricted records, I prefer an approved local workflow. Even then, backups and output filenames matter. Local processing doesn't protect you from overwriting the source or deleting meaningful repeats. My final check is practical: can I explain what disappeared, why it disappeared, and how to recover the original if I chose the wrong rule?
Questions people ask
How can I remove duplicate lines without changing the original order?
I choose a keep-first, order-preserving method and test it with separated repeats before using the full list. For a local command-line option, awk '!seen[$0]++' input.txt > output.txt keeps each exact line's first occurrence. I avoid sort -u when the original order matters because sorting changes the sequence.
How do I remove only consecutive duplicate lines?
I use GNU uniq when I want adjacent repeated lines collapsed while separated repeats remain. For example, uniq input.txt > output.txt writes to a separate result file. I don't sort first for this task, because sorting brings separated repeats together and changes which repeats the command can collapse.
How can I remove both copies of a repeated line?
I look for an exactly-once operation rather than ordinary deduplication. Google Sheets UNIQUE has an exactly_once argument for returning entries that appear only once. I test a sample containing a repeated value and an unrepeated value, because keeping one copy of everything would produce a different result.
Can I ignore capitalization when checking for duplicate lines?
Yes, but I make that an explicit choice rather than assuming it. GNU uniq -i ignores case while still requiring adjacent matches. Google Sheets Remove duplicates also treats capitalization differences as duplicates. If I normalize case in a separate preparation step, I keep the original because the output's spelling has changed.
What is the difference between uniq and sort -u?
I think of uniq as an adjacent-repeat filter and sort -u as a sorting operation that also returns unique lines. GNU describes sort -u as equivalent to sort followed by uniq in the standard case, but not with every sort option. Neither is my default for preserving first-appearance order across separated repeats.
Can I safely deduplicate a large text file without overwriting the original?
I keep an untouched backup and write the cleaned result to a different filename. Then I inspect the output before replacing anything. For large files, I choose a method suited to the available system rather than assuming a browser has unlimited capacity. I never redirect a command's output into its own input file.
What to do next
I start with a copy, decide what counts as a match, and keep the first occurrence unless the task calls for something else. You can do the same now: open Remove Duplicate Lines, try a short sample, then paste your plain-text list and review the cleaned result. Check the order, blank lines and any near-matches before saving it separately. If your data has related spreadsheet columns or quoted CSV records, pause and use a record-aware method instead. A useful cleanup removes unwanted repetition without quietly removing information you still need.
Sources I checked
- GNU Coreutils Manual: Operating on sorted files
- GNU Coreutils Manual: Uniquify
- Google Docs Editors Help: UNIQUE function
- Google Docs Editors Help: Split text, remove duplicates, or trim whitespace
- Microsoft Support: Filter for unique values or remove duplicate values
- Microsoft Learn: Working with duplicate values