Pipes
Processes
COP-3402
Table of Contents
Pipe basics
- What is a pipe?
- How can we use a pipe in Bash?
Pipes are a form of inter-process communication. They are a tool sending information from one process to another.
We can use a pipe in Bash to send the stdout of one process to the stdin of another.
For example, we can send the output of find /usr/include to grep '\.h' like so:
find /usr/include/ | grep '\.h'
But how do pipes actually work?
A pipe is a virtual file
- A pipe has two ends.
- We can read from one and write to the other.
- What abstraction does this satisfy?
Pipes satisfy the file abstraction, because we can read bytes from and write bytes to them.
Let's look at drawing [1] of the board notes. A pipe can be viewed just the same as pipe in the physical world, a cylinder with two ends. One end is the read end, and other is the write end.
A pipe "connects" two processes
- What file descriptors do processes start with?
- How can we connect them with pipes?
Processes begin with three open file descriptors: stdin, stdout, and stderr.
These are illustrated in drawing [2], with both find and grep receiving their own copies of these descriptors.
How can we change these file descriptors, so that find and grep communicate via the pipe?
We can redirect the stdout of find to the write end of the pipe, and the stdin of grep to the read end of the pipe.
This is shown in drawing [3].
You might be wondering, why can't we just redirect the stdout of find to the stdin of grep directly?
Why we need pipes
- How does redirection affect the file entry table?
Redirection changes a file descriptor to point to a different entry in the file entry table, which is shared system-wide across processes.
So we can't just redirect find's stdout to grep's stdin, because this would redirect find's output to the default stdin of all processes, which could lead to unexpected results.
This is shown in drawing [4].
So how can we use pipes in our code?
- With (you guessed it) a system call.
man 2 pipe.
pipe_basic.c
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
// Create an enum for easily referring to
// read and write ends.
enum { READ = 0, WRITE = 1 };
int main(void) {
int pipefd[2];
char message[] = "Hello, world!\n";
size_t count = strlen(message);
char result[16];
// Create the pipe.
if (-1 == pipe(pipefd)) {
perror("pipe");
exit(1);
}
// Write to the write-end.
write(pipefd[WRITE], message, count);
close(pipefd[WRITE]);
// Read from the read-end.
read(pipefd[READ], result, count);
close(pipefd[READ]);
// Write to stdout.
write(STDOUT_FILENO, result, count);
return 0;
}
The above program opens a pipe with pipe(), writes some bytes to it, reads them, then writes the result to stdout.
Note that it also closes each end of the pipe after reading from or writing to it; this is to maintain the good practice of closing file descriptors after we are done with them.
The connection between this program and the pipe is illustrated in drawing [5].
How can we use pipes to connect a parent and a child process?
pipe_fork.c
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/wait.h>
#include <unistd.h>
enum { READ = 0, WRITE = 1 };
int main(void) {
int pipefd[2];
char message[] = "Hello from parent\n";
size_t count = strlen(message);
char result[16];
pipe(pipefd);
switch(fork()) {
case 0:
// Close write end, child only reads
close(pipefd[WRITE]);
read(pipefd[READ], result, count);
// Close read end once finished.
close(pipefd[READ]);
write(STDOUT_FILENO, result, count);
fflush(stdout);
_exit(0);
default:
// Close read end, parent only writes
close(pipefd[READ]);
write(pipefd[WRITE], message, count);
// Close write end once finished.
close(pipefd[WRITE]);
wait(NULL);
exit(0);
}
}
In the above example, the parent process first opens a pipe with pipe(), then invokes the fork() system call to create a child process.
The child inherits a copy of all the parent's open file descriptors, including the read and write ends of the pipe.
After invoking fork(), the parent writes a message to the child, then wait()~s for the child to finish.
Meanwhile, the child reads the message from the read end of the pipe, and writes the result to stdout.
Throughout the entire program, the parent and the child ~close() pipe file descriptors when they are done using them.
This process is shown in drawing [6].
Now how about implementing a pipeline like we would write in Bash.
filter.c
Goal: Implement find /usr/include | grep '.\h'
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/wait.h>
#include <unistd.h>
// find /usr/include | grep '\.h'
enum { READ = 0, WRITE = 1 };
int main(void) {
int pipefd[2];
pipe(pipefd);
switch(fork()) {
case 0:
// Close read end since left child only
// writes.
close(pipefd[READ]);
dup2(pipefd[WRITE], STDOUT_FILENO);
close(pipefd[WRITE]);
execlp("find", "find", "/usr/include/", NULL);
// Kernel will close write end for us after
// execlp() finishes.
_exit(1);
}
switch(fork()) {
case 0:
// Close write end because right child only
// reads.
close(pipefd[WRITE]);
dup2(pipefd[READ], STDIN_FILENO);
close(pipefd[READ]);
execlp("grep", "grep", "\\.h", NULL);
// Kernel will close read end for us after
// execlp() finishes.
_exit(1);
}
// Close open file descriptors in parent.
// What if we forget to do this?
close(pipefd[READ]);
close(pipefd[WRITE]);
wait(NULL);
wait(NULL);
exit(0);
}
The above program creates not one, but two children to pipe the output of find /usr/include to grep '.\h'.
The parent first creates a pipe with pipe(), so that its read and write file descriptors can be inherited by both its children.
Then, it creates a child process with fork(), redirects that child's stdout to the write end of the pipe with dup2(), and finally uses execlp() to replace the process with find.
Next, the parent creates a second child process, redirects its stdin to the read end of the pipe, and finally replaces the second child with grep.
Finally, the parent invokes wait() twice to wait for the children to finish.
What happens if we forget to close the read and write ends of the pipe in the parent? The program will not terminate, but hang, because without closing the write end of the pipe, the right child process will continue to wait for more input.
Drawing [7] illustrates how this program works.