Remove Duplicate Lines Online
Clean repeated lines from a list without overwriting your original. Keep the first or last occurrence, adjust matching rules and copy the cleaned result in its retained order.
Maximum 500,000 UTF-16 code units. The original text is preserved. No input is silently truncated.
Matching options do not change the spelling or outer spaces of retained lines. Output line endings use LF.
Loading the tool…
A final line ending terminates the last line; it does not add another counted line. When present in the input, a final line ending is retained if any lines survive. Blank lines removed by the blank-line option are counted separately from duplicates.
How to remove duplicate lines
1. Paste your list
Put one item on each line in the original-text box.
2. Choose matching rules
Keep the first or last copy and decide whether case or outer spaces matter.
3. Clean and export
Click Remove duplicates, review the counts, then copy or download the result.
Clean repeated entries without sorting your list
This duplicate line remover compares complete lines, including repeated entries that are not next to each other. It can help clean lists of labels, identifiers, keywords or other plain-text entries while preserving the order of the retained occurrences.
Keep the first or last occurrence
Keeping the first occurrence retains the earliest matching line and removes later copies. Keeping the last occurrence retains the latest matching line. The remaining lines stay in their original relative order; the tool does not alphabetize them.
For example, the lines Apple, Banana, Apple produce Apple then Banana when keeping the first copy. Keeping the last copy produces Banana then Apple, because those are the positions of the retained lines.
Compare without changing retained text
By default, case and whitespace are significant. Apple and apple are different, and a line with surrounding spaces differs from one without them. Ignore letter case uses English locale lowercasing for comparison. It is not full linguistic Unicode case folding, so some language-specific equivalents may remain distinct.
Ignoring leading and trailing whitespace applies JavaScript trimming only to the comparison key. It does not remove spaces from the retained line or collapse spaces inside it. Visually identical Unicode strings with different internal representations are not automatically normalized.
Choose how to handle blank lines
With the blank-line option off, blank lines are compared like other lines. Identical blank lines are deduplicated. Lines containing different whitespace can remain distinct unless outer-whitespace matching is enabled.
With the option on, empty and whitespace-only lines are removed before duplicate matching. They appear in the separate blank-line counter, so input lines equal retained lines plus duplicate removals plus separately removed blank lines.
Plain-text matching has limits
The tool compares entire lines, not columns or records. It does not parse CSV quoting, validate email addresses or detect similar spellings. Different URLs or differently formatted identifiers stay distinct unless they match under your selected options.
Output uses LF line endings. The tool recognizes CRLF, CR and Unicode line separators as line boundaries, so it is not a byte-preserving file editor. It leaves the original input available for review.
Use Text Compare to inspect the changes, or the Word Counter to analyze the cleaned text.
Privacy and local processing
This tool’s code processes your list in your browser. It does not send your text to an API or save it in browser storage. Clipboard access happens when you request a copy, and TXT downloads are generated locally in UTF-8.
Your browser, extensions and other scripts installed on the site have their own behavior, which this tool does not control.
Frequently asked questions
Does this remove all copies of repeated lines?
No. It retains one occurrence of each matching line, according to your first-or-last selection, and removes the other copies.
Are repeated lines removed only when adjacent?
No. Matching lines are deduplicated throughout the entire input, even when other entries appear between them.
Does ignoring outer whitespace trim the result?
No. Trimming is used only to decide whether lines match. The retained line keeps its original content and spacing.
Does the tool support Arabic and accented letters?
Yes, the input and output support Unicode text. Exact matching distinguishes diacritics and different Unicode representations. The optional case comparison uses English locale lowercasing.
Why is a trailing newline not an extra input line?
This tool treats a final newline as the terminator of the last line. Additional empty lines before that final terminator are counted. Empty input has zero lines.
Can I edit the cleaned result?
The result is read only so its statistics remain consistent. Copy or download it to edit elsewhere. Changing the source or settings clears the old result until you process the list again.