Sort & Dedupe Lines

A grab bag for the things you'd do with sort | uniq | grep -v '^$' in a shell — alphabetical, descending, or natural sort (so 'item10' comes after 'item2', not after 'item1'); deduplicate (case-sensitive or not); trim whitespace; drop blank lines; reverse the result; or shuffle for a randomized order. Counts shown for input, output, and removed lines so you can see what changed.

Natural sort is the option worth knowing about, because plain alphabetical order treats a number as a string of digits. item2 sorts after item10 there, since the comparison stops at the first character that differs and 1 precedes 2. Natural sort reads runs of digits as numbers and puts them in the order a person expects, which matters for filenames, version strings and anything ending in a counter.

Alphabetical order is not one thing either. Sorting by code point puts every uppercase letter before every lowercase one, so Zebra lands above apple, and accented characters fall outside the basic range entirely. A locale-aware comparison folds case and treats é next to e, which is what a reader expects and what a naive sort does not do. The case-insensitive toggle here addresses the first half of that.

Deduplication keeps the first occurrence and runs before sorting, so the order of the input decides which duplicate survives. That is invisible when the lines are identical and matters when they differ only in case or in trailing whitespace: trimming before deduplicating collapses foo and foo into one entry, while doing it the other way round leaves both. The order of operations is doing real work.

The counts are the part to read. A dedupe that removes far more lines than you expected usually means whitespace or case was hiding duplicates you did not know about, and one that removes none when you expected some means the opposite.