Remove Duplicate Lines
One of each, in the order they first appeared.
One of each, in the order they first appeared.
How to use it
- Paste your list. One item per line. Windows and Mac line endings are handled the same.
- Decide about case. By default Apple and apple count as the same line. Turn on matching to keep both.
- Copy the result. The line underneath says how many were removed, so you can sanity check the count.
When you would use this
Deduplicating a list is the sort of job that is trivial in a shell and irritating everywhere else. Pasting into a spreadsheet, sorting and eyeballing works until the list is long enough that you stop trusting your eyes. Two decisions matter here and both are exposed rather than assumed. The first is which copy survives: this keeps the earliest, so an order that meant something is preserved rather than scrambled. The second is case. Treating Apple and apple as the same thing is right for names and email addresses and wrong for identifiers and passwords, so it is a switch rather than a rule. Whitespace is left alone deliberately, because a trailing space really is a different line, and quietly trimming would hide a data problem you probably want to see.
Questions
- Which duplicate is kept?
- The first one. The order of what remains is the order it first appeared in, which matters when the list is ranked or chronological.
- Does it ignore leading and trailing spaces?
- No. A line with a trailing space is genuinely a different line. Run it through the trim tool first if you want those treated as equal.
- Is my list sent anywhere?
- No. The work happens in the page. There is no server here, which is also why it keeps going with the network disconnected.