Use named pipe linux

Fun with Unix Named Pipes

Named pipes are very useful Linux feature that enables quick and cheap inter-process communication. A named pipe is a special type of file within the file system that behaves like the traditional pipes in Unix system.

Features of named pipes

Some characteristics of names pipes and their differences over regular files or Unix sockets:

Both reader & writer processes need to open the file simultaneously, otherwise opening of file for read/write operations will be blocked.

It has to be open at both ends simultaneously before you can proceed to do any input or output operations on it. Opening a FIFO for reading normally blocks until some other process opens the same FIFO for writing, and vice versa. [1]

The FIFO special file has no contents on the file system; the file system entry merely serves as a reference point so that processes can access the pipe using a name in the file system.[2]

Some fun uses of named pipes:

Real-time telecooperation via terminal:

Using script and mkfifo we can share our terminal activity in real-time with other user.

Bi-directional communication

Be careful when opening a named pipe with both read and write flag (e.g. r+ ) from the same process. It doesn’t enable bi-directional pipe but would actually cause a deadlock and loopback data within the same process. Consider the following example that doesn’t block and wait for other process to read from the named pipe. Instead it pipes the data to the same process:

// Bad example that causes deadlock const fs = require('fs') // open in both read & write mode // isn't blocked for other process to open the pipe const fd = fs.openSync('./pipe', 'r+') for(let i=0;i10;i++) const input = 'hello world!'; console.log('sending:', input) fs.writeSync(fd, input) const b = new Buffer(1024) fs.readSync(fd, b, 0, b.length) console.log('received:', b.toString()) > 

To avoid this deadlock, we can open separate file handlers with read-only and write-only flags.

const fs = require('fs') const input = 'hello world!'; console.log('sending:', input) const fw = fs.openSync('./pipe', 'w') // blocked until a reader attached fs.writeSync(fw, input) console.log('sent!') console.log('waiting for reply') const fr = fs.openSync('./pipe', 'r') // blocked until a writer attached const b = new Buffer(1024) fs.readSync(fr, b, 0, b.length) console.log('received:', b.toString()) 

const fs = require('fs') const fr = fs.openSync('./pipe', 'r') // blocked until writer attached const b = new Buffer(1024) fs.readSync(fr, b, 0, b.length) console.log('received:', b.toString()) setTimeout(()=> console.log('sending:', b.toString().toUpperCase()) const fw = fs.openSync('./pipe', 'w') //blocked until reader attached fs.writeSync(fw, b.toString().toUpperCase()) >, 3000) 

Transfer large files between processes without temporary files

A named pipe can be used to transfer large amount data from one application to another without the use of an intermediate temporary file. This is useful if either of the processes doesn’t support anonymous piping (e.g. stdin/stdout pipes).

$ mkfifo -m 0666 /tmp/pipe $ gzip -d < file.gz > /tmp/pipe $ mysql -e "LOAD DATA INFILE '/tmp/pipe' INTO TABLE t1" db1 

Random piping to multiple simultaneous readers

If multiple processes try to read from the same pipe, the kernel seems like randomly selects a reader and pipes to it. So it’s not possible to broadcast the same data to multiple readers simultaneously. For example try different combinations of following reader.sh and writer.sh scripts (e.g multiple readers, multiple writers) and notice the output.

#!/bin/bash if [ ! -p pipe ];then mkfifo pipe fi col="$(( $RANDOM * 6 / 32767 + 1 ))" while true do i=$((i+1)) echo -e "\e[0;3$col>m$i" sleep 1 done > pipe 

Источник

named pipes в Unix

Я давно читал про них, ещё когда учился основам юникс, но как-то не было нужды с ними работать. И, вот, нужда возникла.

Некая программа (допустим, foo) не умеет писать вывод в stdout, только в файл. Даже «-» в качестве имени файла всего лишь создаёт файл с названием «-» [большинство умных программ под unix знают, что одиночный минус вместо имени файла означает вывод в stdout]. Аналогично она отвергает и /dev/stdout.

Другая же программа, обрабатывающая результаты первой, допустим, bar, читает из stdin и пишет в stdout. (если быть точным, первое — это трейсер специального вида, дающий двоичный дамп, а второе — конвертор, печатающий их же в человекочитаемом виде).

Нужно их объединить в конвеер.

Некрасивый вариант — использование обычного файла. Записал, прочитал.

Есть куда более красивый вариант — это именованные пайпы. Так как у пайпа есть имя, мы можем передать его как файл первой программе, а потом передать содержимое другой.

mkfifo mypipe cat mypipe | bar & foo mypipe& rm mypipe

Пайп в файловой системе выглядит так:

ls -l mypipe prw-r--r-- 1 root root 0 May 24 12:23 mypipe

(акцент на букву ‘p’ первым символом).

Как это работает? Фактически, fifo aka named pipe — это «обыкновенный pipe», примерно такой, который кодируется палкой «|». Однако, у него есть ИМЯ и это имя можно указывать всюду, где требуется файл.

Программа, которая пишет в именованный пайп, ведёт себя с ним как с файлом. Т.е. пишет себе и пишет. Программа, которая читает — аналогично. Читает себе и читает. Чтение идёт в том порядке, как была осуществлена запись (FIFO — first in first out). Положения относительно пайпа (слева/справа) определяются тем, кто читает, а кто пишет.

Важная же особенность пайпа — способность тормознуть читающую/пищущую программу, если буфер пуст/переполнен.

Рассмотрим на примере чтения. Программа пишет в пайп одну строчку в секунду. Программа чтения читает с максимально возможной скоростью. Программа «вычитывает» всё, что было в буфере, и посылает следующий запрос. Ядро этот запрос задерживает до того момента, пока не появятся данные. Таким образом, можно не париться с синхронизацией — появятся данные, программа-обработчик получит управление обратно из read() и обработает очередную порцию данных.

Источник

Anonymous and Named Pipes in Linux

In Linux, a pipe is a mechanism that allows the output of one command to be used as the input for another command. Pipes allow for powerful command line operations by allowing the output of one command to be used as input for another command.

Pipes

Pipes are a feature in Linux and other Unix-like operating systems that allow the output of one command to be passed as input to another command. They are represented by the «|» symbol, and are often used to chain multiple commands together to perform complex tasks.

For example, the command «ls -l | grep ‘^d'» will list all directories in the current directory, because the output of the «ls -l» command is being used as the input for the «grep ‘^d'» command, which filters the list to only show lines that begin with the letter «d».

Pipelines in Bash

Pipelines are a feature in Bash, which is the default shell on most Linux and Unix-like operating systems. They allow you to chain together multiple commands in a single line of code, where the output of one command is passed as input to the next command in the pipeline. The pipeline operator in Bash is the «|» symbol.

For example, you can use the command «ls -l | grep ‘^d'» to list all directories in the current directory. The «ls -l» command will list all files in the current directory in long format, and the output of that command is passed as input to the «grep ‘^d'» command, which filters the list to only show lines that begin with the letter «d», which corresponds to directories.

Another example is using the command «cat file1.txt | sort | uniq > file2.txt» to sort and remove duplicates of the content of file1 and save the result into file2. The cat command is used to read the content of file1.txt, then sort command will sort the content and uniq command will remove the duplicate lines and the final result is saved into file2.txt by > operator

Pipelines can be used to perform complex operations with just a few commands, and they are a powerful feature of Bash that can help you automate and streamline your work on the command line.

Named Pipes

A named pipe, also known as a FIFO (first-in, first-out) file, is a special type of file that allows two or more processes to communicate with each other by sending and receiving data through the pipe. A named pipe is created using the «mkfifo» command, and can be used like a regular file for reading and writing data. However, unlike a regular file, a named pipe does not reside on a storage device, but instead acts as a buffer for inter-process communication.

Once a named pipe is created, one process can write data to it, while another process can read data from it. The data is read in the order it was written, hence the name «first-in, first-out». This allows processes to communicate with each other without the need for explicit message passing or shared memory.

For example, you can use named pipe to send the output of a long running command to another command that process the output in real-time.

command1 | tee >(command2) | command3

Named pipes are useful when two or more processes need to communicate with each other, but do not have a direct way to do so. They can be used for inter-process communication, data transfer, and even for redirecting one command’s output to another in real-time.

Temporary Named Pipes

Temporary named pipes are a type of named pipes that are created for a specific purpose and are deleted after they are no longer needed. They are typically used for short-term inter-process communication, where the processes involved do not need to maintain a persistent connection. Unlike a regularly named pipe, a temporarily named pipe is not created with the «mkfifo» command and does not have a name associated with it. Instead, it is created automatically when it is needed and deleted once it is no longer in use.

One common use case for temporarily named pipes is in shell scripts. Temporary named pipes can be used to pass data between commands within a script, without the need for temporary files. The process that writes to the pipe is called the producer and the process that reads from it is called the consumer. They can be used as a replacement for input/output redirection in shell scripts.

For example, a command like «command1 | tee >(command2) | command3» will create a temporarily named pipe and pass the output of command1 to command2 and command3.

Temporary named pipes are a useful feature in Linux and Unix-like operating systems, as they allow for efficient inter-process communication without the need for persistent named pipes or temporary files.

When to Use Named or Anonymous Pipes?

Named pipes and anonymous pipes are both used for inter-process communication (IPC) in Linux and Unix-like operating systems, but they have different use cases and characteristics.

Named pipes, also known as FIFOs, are useful when you need to maintain a persistent connection between processes for an extended period of time. They are created with a unique name, and can be used by multiple processes to send and receive data. Named pipes can be used for data transfer between processes that are running on the same machine, or even between different machines over a network. They can be used for long-term IPC, where multiple processes need to communicate with each other over an extended period of time.

Anonymous pipes, on the other hand, are created automatically when they are needed, and are deleted when they are no longer in use. Anonymous pipes are useful when you need to pass data between processes within a single command or script, and you do not need to maintain a persistent connection. Anonymous pipes are used for short-term IPC, where the processes involved do not need to maintain a persistent connection. Anonymous pipes are also known as temporary named pipes, and they’re commonly used as a replacement for input/output redirection in shell scripts.

Conclusion

Pipes and named pipes are both features of Linux and Unix-like operating systems that allow for inter-process communication (IPC). Pipes, represented by the «|» symbol, allow the output of one command to be passed as input to another command, allowing for powerful command-line operations and data manipulation. Named pipes, also known as FIFOs, are useful when you need to maintain a persistent connection between processes for an extended period of time. They are created with a unique name and can be used by multiple processes to send and receive data. Anonymous pipes, on the other hand, are created automatically when they are needed, and are deleted when they are no longer in use.

Источник