Pipe Overview
The pipe symbol '|' in Linux/Unix shells serves to pass the output of the command on the left as input to the command on the right. This allows users to combine multiple commands to build powerful and flexible data processing pipelines.
How Pipes Work
The standard output (stdout) of the first command is stored in a temporary buffer created by the shell, and the contents of this buffer are then passed as the standard input (stdin) to the second command. This process can connect multiple commands to form a pipeline.
Characteristics of Pipes
- Unidirectional Flow: Data always flows from left to right, in one direction only.
- Data Reuse: The result of a preceding command can be immediately used as input for the next command.
- Modularity: Large and complex tasks can be broken down into smaller, simpler commands for processing.
- Efficiency: Data is processed in memory without the need to create temporary files, making it efficient.
Pipe Usage Examples
Experience the power of pipes through practical examples that combine multiple commands.
Search for a Specific Pattern in File Listings
ls -l | grep jsonOutputs the file list of the current directory using `ls -l`, then uses `grep` to search only for lines containing the string 'json'. This pipeline passes the output of `ls -l` as input to `grep`.
Find a Specific Process in the Process List
ps aux | grep nginx | grep -v grepChecks the list of all processes with `ps aux`, then uses `grep` to filter only 'nginx' processes. Since `grep` itself is a process, the `grep nginx` process should be excluded.
Find the 5 Largest Files
ls -lh | sort -rh | head -5Outputs a detailed file list with `ls -lh`, then sorts by size (`-h` option for human-readable units) in reverse order (`-r`) using `sort -rh`. Finally, `head -5` outputs only the top 5 results.
Count .txt Files in the Current Directory
ls | grep '.txt' | wc -lOutputs a file list with `ls`, filters only files with the '.txt' extension using `grep`. Finally, `wc -l` counts the number of filtered lines (`-l` option) to get the total number of files.
Delete Files by Combining Pipe with xargs
find . -name '*.log' | xargs rmAfter finding files with a specific pattern using the `find` command, `xargs` is used to pass the found file list as arguments to the `rm` command for deletion. Pipes alone make it difficult to pass arguments to `rm`, hence the use with `xargs`.
Difference from Redirection
Redirection, often used with pipes, is similar in controlling data flow but has significant differences in its method.
Pipe vs. Redirection
- Pipe (`|`): Connects the **output** of one command to the **input** of another.
- Redirection (`>`, `>>`): Sends the **output** of a command to a file, or sends the content of a file as **input** to a command.
Redirection Example
ls -l > file_list.txtSaves the result of `ls -l` to the `file_list.txt` file. This is not used as input for another command but is written directly to the file.