Woodstock Blog

a tech blog for general algorithmic interview questions

[Design] How Is Pipe Implemented in Unix/Linux

Overview

In Unix-like OS, a pipeline is a set of processes chained by their standard streams, so that the output of each process (stdout) feeds directly as input (stdin) to the next one.

Pipes are unidirectional byte streams which connect the standard output from one process into the standard input of another process. Neither process is aware of this redirection and behaves just as it would normally. It is the shell which sets up these temporary pipes between the processes.

Process

  1. Linux has a VFS called pipefs that is mounted in the kernel (not in user space)

    PipeFS is a unique virtual filesystem. This filesystem is mounted inside the kernel rather than in the userspace. While most filesystems are mounted under “/”, PipeFS is mounted on “pipe:”, making PipeFS its own root (yes, a second root filesystem).

    This filesystem is one superblock and cannot exceed that amount system-wide. The entry point of this filesystem/second-root is the system-call “pipe()”. Unlike the other virtual/pseudo filesystems, this one cannot be viewed.

    Many of you may be wondering what purpose this PipeFS filesystem serves. Unix pipes use this filesystem. When a pipe is used (eg. ls | less), the pipe() system-call makes a new pipe object on this filesystem. Without this filesystem, pipes cannot be made.

    Also, threads and forks communicate together via pipes. Without PipeFS, processes could not fork and threads could not communicate.

    Network pipes also rely on this virtual/pseudo filesystem.

  2. pipefs has a single super block and is mounted at it’s own root (pipe:), alongside /

  3. pipefs cannot be viewed directly unlike most file systems

  4. The entry to pipefs is via the pipe(2) syscall

  5. The pipe(2) syscall used by shells for piping with the | operator (or manually from any other process) creates a new file in pipefs which behaves pretty much like a normal file

  6. The file on the left hand side of the pipe operator has its stdout redirected to the temporary file created in pipefs

  7. The file on the right hand side of the pipe operator has its stdin set to the file on pipefs

  8. pipefs is stored in memory and through some kernel magic

Ref: http://unix.stackexchange.com/q/148401