What the flip are Unix Named Pipes?
In Unix systems, pipes and redirections are the bridges that join our programs. They are an underrated resource that can aid developers in their own work in powerful ways. Today’s post will explore pipes from their humble beginnings to how we use them in our programs (with example help from Golang).
Humble beginnings
Let’s start with the humble pipe operator | . If you are running a Unix shell, we can explore the pipe operator with some simple examples:
> echo "Hello friends" Hello friends > echo "Hello friends" | tr "[:lower:]" "[:upper:]" HELLO FRIENDS
In this short example, we are echoing «Hello friends» and using a pipe to pass that output to the translate characters program tr and taking all the lower case letters and upper casing them!
If you are unfamiliar with tr , run man tr to see more. The description includes information on [:class:] usage.
In fact, we could add more pipes if we wanted to «glue» together more programs. We could even just do that with passing out put back to translate again!
> echo "Hello friends!" | tr "[:upper:]" "[:lower:]" | tr "[:space:]" "\n" hello friends
You get the gist. What gets cool though is that you can create «named pipes» to start piping in some cooler ways.
Named pipes
In computing, a named pipe (also known as a FIFO for its behavior) is an extension to the traditional pipe concept on Unix and Unix-like systems, and is one of the methods of inter-process communication (IPC).
For clarification, FIFO in this context stands for «first-in, first-out». We now also understand that it is an extension on tradition pipes, so how do we make one? Luckily, we have a program mkfifo . In terminal, we can run this and se that a named pipe will be created in the current working directory.
> mkfifo example-pipe > ls example-pipe
If were now to redirect anything to this named pipe ie ls > example-pipe , the program will wait for something for the named pipe to pass to.
In another terminal, we can do that very things by running cat < example-pipe to see the output in that terminal. Neat!
Running the output of a named pipe to Golang
Now, let’s see how this can be applied to our programs. Here, I am going to use a Golang program to demonstrate how we can do this. I won’t dive too deep into the code (Go has the amazing go doc
package main import ( "os" "os/signal" "syscall" "path/filepath" "bufio" "sync" "log" "fmt" "time" ) func main() < // Setup our Ctrl+C handler SetupCloseHandler() ReadPipe() >func ReadPipe() < fmt.Println("Starting pipe") p := "example-pipe" dir, err := os.Getwd() if (err != nil) < log.Fatal(err) panic("Could not read working directory") >namedPipe := filepath.Join(dir, p) stdout, _ := os.OpenFile(namedPipe, os.O_RDONLY|syscall.O_NONBLOCK, 0600) syscall.SetNonblock(int(stdout.Fd()), false) // 128MB buffer bufSize := 1000000 * 128 mutex := &sync.Mutex<> for < scanner := bufio.NewScanner(bufio.NewReaderSize(stdout, bufSize)) for scanner.Scan() < mutex.Lock() fmt.Println(scanner.Text()) mutex.Unlock() >time.Sleep(100 * time.Millisecond) > defer stdout.Close() > // SetupCloseHandler creates a 'listener' on a new goroutine which will notify the // program if it receives an interrupt from the OS. We then handle this by calling // our clean up procedure and exiting the program. func SetupCloseHandler() < c := make(chan os.Signal) signal.Notify(c, os.Interrupt, syscall.SIGTERM) go func() < () >
- SetupCloseHandler that creates a channel to listen out for our POSIX standard interrupt signal to exit the program.
- ReadPipe that creates an infinite loop in the program to read from the «example-pipe» every 100ms.
Author’s note: ReadPipe and SetupCloseHandler did not need to be public functions, I am just copying them across from other packages in my program that did require it to be so.
If we run go run main.go in one terminal and echo «test» > example-pipe into another, you can see that we get the output through the pipe in our Go program!
Since we are indefinitely checking the pipe until the terminate signal is passed, we can run whatever we want into that example-pipe and see it coming out through our program.
Of course, once we run our terminate signal, we can close the Go program (and you can see Go handling that signal).
Handling the terminate signal
Awesome! We’ve managed to create some named pipes and use content piped into them in our program!
Conclusion
This has been a short look at pipes and tying them into our programs. These become very useful when you are building tools to glue others together.
For example at work, we have a tool that I am about to take ownership of which connects together our developer experience from running our different services to migrations, installations and more. Named pipes is our way of being able to selectively pipe logs from each of those running services into one place to empower the developer experience and ensure that they are not overwhelmed!
I have linked a bunch of great resources below that relate to more information on both the pipes and everything that has been happening on the Golang side of things.
Anonymous and Named Pipes in Linux
In Linux, a pipe is a mechanism that allows the output of one command to be used as the input for another command. Pipes allow for powerful command line operations by allowing the output of one command to be used as input for another command.
Pipes
Pipes are a feature in Linux and other Unix-like operating systems that allow the output of one command to be passed as input to another command. They are represented by the «|» symbol, and are often used to chain multiple commands together to perform complex tasks.
For example, the command «ls -l | grep ‘^d'» will list all directories in the current directory, because the output of the «ls -l» command is being used as the input for the «grep ‘^d'» command, which filters the list to only show lines that begin with the letter «d».
Pipelines in Bash
Pipelines are a feature in Bash, which is the default shell on most Linux and Unix-like operating systems. They allow you to chain together multiple commands in a single line of code, where the output of one command is passed as input to the next command in the pipeline. The pipeline operator in Bash is the «|» symbol.
For example, you can use the command «ls -l | grep ‘^d'» to list all directories in the current directory. The «ls -l» command will list all files in the current directory in long format, and the output of that command is passed as input to the «grep ‘^d'» command, which filters the list to only show lines that begin with the letter «d», which corresponds to directories.
Another example is using the command «cat file1.txt | sort | uniq > file2.txt» to sort and remove duplicates of the content of file1 and save the result into file2. The cat command is used to read the content of file1.txt, then sort command will sort the content and uniq command will remove the duplicate lines and the final result is saved into file2.txt by > operator
Pipelines can be used to perform complex operations with just a few commands, and they are a powerful feature of Bash that can help you automate and streamline your work on the command line.
Named Pipes
A named pipe, also known as a FIFO (first-in, first-out) file, is a special type of file that allows two or more processes to communicate with each other by sending and receiving data through the pipe. A named pipe is created using the «mkfifo» command, and can be used like a regular file for reading and writing data. However, unlike a regular file, a named pipe does not reside on a storage device, but instead acts as a buffer for inter-process communication.
Once a named pipe is created, one process can write data to it, while another process can read data from it. The data is read in the order it was written, hence the name «first-in, first-out». This allows processes to communicate with each other without the need for explicit message passing or shared memory.
For example, you can use named pipe to send the output of a long running command to another command that process the output in real-time.
command1 | tee >(command2) | command3
Named pipes are useful when two or more processes need to communicate with each other, but do not have a direct way to do so. They can be used for inter-process communication, data transfer, and even for redirecting one command’s output to another in real-time.
Temporary Named Pipes
Temporary named pipes are a type of named pipes that are created for a specific purpose and are deleted after they are no longer needed. They are typically used for short-term inter-process communication, where the processes involved do not need to maintain a persistent connection. Unlike a regularly named pipe, a temporarily named pipe is not created with the «mkfifo» command and does not have a name associated with it. Instead, it is created automatically when it is needed and deleted once it is no longer in use.
One common use case for temporarily named pipes is in shell scripts. Temporary named pipes can be used to pass data between commands within a script, without the need for temporary files. The process that writes to the pipe is called the producer and the process that reads from it is called the consumer. They can be used as a replacement for input/output redirection in shell scripts.
For example, a command like «command1 | tee >(command2) | command3» will create a temporarily named pipe and pass the output of command1 to command2 and command3.
Temporary named pipes are a useful feature in Linux and Unix-like operating systems, as they allow for efficient inter-process communication without the need for persistent named pipes or temporary files.
When to Use Named or Anonymous Pipes?
Named pipes and anonymous pipes are both used for inter-process communication (IPC) in Linux and Unix-like operating systems, but they have different use cases and characteristics.
Named pipes, also known as FIFOs, are useful when you need to maintain a persistent connection between processes for an extended period of time. They are created with a unique name, and can be used by multiple processes to send and receive data. Named pipes can be used for data transfer between processes that are running on the same machine, or even between different machines over a network. They can be used for long-term IPC, where multiple processes need to communicate with each other over an extended period of time.
Anonymous pipes, on the other hand, are created automatically when they are needed, and are deleted when they are no longer in use. Anonymous pipes are useful when you need to pass data between processes within a single command or script, and you do not need to maintain a persistent connection. Anonymous pipes are used for short-term IPC, where the processes involved do not need to maintain a persistent connection. Anonymous pipes are also known as temporary named pipes, and they’re commonly used as a replacement for input/output redirection in shell scripts.
Conclusion
Pipes and named pipes are both features of Linux and Unix-like operating systems that allow for inter-process communication (IPC). Pipes, represented by the «|» symbol, allow the output of one command to be passed as input to another command, allowing for powerful command-line operations and data manipulation. Named pipes, also known as FIFOs, are useful when you need to maintain a persistent connection between processes for an extended period of time. They are created with a unique name and can be used by multiple processes to send and receive data. Anonymous pipes, on the other hand, are created automatically when they are needed, and are deleted when they are no longer in use.