uniq Command Overview
`uniq` is short for 'unique', and it performs the function of finding and processing duplicate lines within a file. The crucial point here is that `uniq` only processes **consecutively duplicated lines**. Therefore, to remove duplicates from an entire file, you must first sort the data using the `sort` command.
How uniq Works
The `uniq` command reads input lines one by one and compares each with the immediately preceding line. If two lines are identical, they are considered duplicates; otherwise, they are considered new unique lines. Because of this process, to remove duplicates from an entire file, it is essential to first sort the file using the `sort` command.
Key Options
By utilizing the various options of the `uniq` command, you can perform detailed tasks such as duplicate removal, counting, and specific line output.
1. Basic Functions
2. Control Comparison Method
Generated command:
Try combining the commands.
Description:
`uniq` Executes the command.
Combine the above options to virtually execute commands with AI.
Commonly Used Examples
Learn how to effectively process data by using `uniq` with `sort`.
Remove Duplicate Lines from Entire File
sort data.txt | uniqSorts `data.txt` with `sort`, then removes duplicate lines from the entire file using `uniq`. This combination is the most common usage.
Output Duplicated Lines with Counts
sort data.txt | uniq -cRemoves duplicate lines from `data.txt` and outputs each line along with the number of times it appeared.
Output Only Duplicated Lines from Entire File
sort data.txt | uniq -dOutputs only lines that appeared more than once in `data.txt`.
Output Only Unique Lines Appearing Once in Entire File
sort data.txt | uniq -uOutputs only lines that appeared exactly once and are not duplicated in `data.txt`.
Remove Duplicates Ignoring Specific Fields
sort log.txt | uniq -f 1Removes duplicates from a log file by ignoring time information (the first field) and comparing only the remaining content.