What are filters in linux

Pipes and filters

I still remember the day, many years ago, when a wise old programmer looked over my shoulder and said, “Ah, Grasshopper, you need a pipe!” and so set me on the path to true enlightenment.

A pipe is a means by which the output from one process becomes the input to a second. In technical terms, the standard output (stout) of one command is sent to the standard input (stdin) of a second command. If you are not sure of the advantages this creates, then let’s look at a simple example.

In this example, we’ll send a directory listing to an email account.

ls -l ~ > ls.tmp
mail -s "Home directory listing" info@markbain-writer.tk < ls.tmp

This works well, but it’s rather cumbersome and requires the creation of an interim file. The use of a pipe allows a simpler command structure and needs no extra files:

ls -l ~ | mail -s "Home directory listing" info@markbain-writer.tk

You will notice that a pipe is defined by the | symbol — not an uppercase i or the number one, but a vertical bar.

Introducing the filter

A pipe can pass the standard output of one operation to the standard input of another, but a filter can modify the stream. A filter takes the standard input, does something useful with it, and then returns it as a standard output. Linux has a large number of filters. Some useful ones are the commands awk , grep , sed , spell , and wc .

If we look back at the our pipe example from above, we see that it gives an output something like:

drwxr-xr-x 4 bainm users 4096 2005-06-05 16:31 Desktop/ drwxr-xr-x 5 bainm users 4096 2004-11-15 00:00 GNUstep/ drwx------ 11 bainm users 4096 2005-06-04 18:02 Mail/ -rw-r--r-- 1 bainm users 10240 2005-01-06 20:36 New_database.kexi drwxr-xr-x 5 bainm users 4096 2005-05-27 12:53 OpenOffice.org1.1.2/ -rwxr-xr-x 1 bainm users 548788 2004-10-20 19:45 Project1* drwxr-xr-x 3 bainm users 4096 2004-10-18 10:52 Projects/ -rw-r--r-- 1 bainm users 4242 2004-10-20 19:45 Unit1.dcu drwxr-xr-x 3 bainm users 4096 2005-05-24 11:59 XamXpm/ drwxr-xr-x 11 bainm users 4096 2005-06-03 10:26 articles/ drwxr-xr-x 2 bainm users 4096 2005-05-30 15:09 backup/

Let’s say that in our email we require only files (not directories) sorted by the largest first and showing only the file name, owner, date last modified, and file size (in that order). To do this, we can use three of the Linux filters: awk (to format), grep (to remove the unwanted lines) and sort (to get the lines in the correct order). In between each filter, we can use a pipe to pass on the result from the individual operations.

The first filter ( grep ) removes any directories from the list by excluding any lines that start with a leading “d”:

The next filter ( awk ) extracts the required fields (file name, user name, access date and time, and file size). It also places the file size at the start line so that the data is ready for sorting:

Obviously, the next filter sorts the data:

And the final filter (another awk ) formats the data ready to be emailed:

Finally, all we have to do is join the filters together with pipes:

The result is something like:

backup.zip bainm 2005-05-30 13:03 1139563 Project1* bainm 2004-10-20 19:45 548788 Delphi_job_spec.rtf bainm 2004-10-14 13:37 217524 output.ps bainm 2004-12-01 21:22 166465 print.pdf bainm 2005-03-06 20:50 47266 kstars.png bainm 2005-03-05 17:35 20586 driving.htm bainm 2004-11-04 21:46 14977 comp.htm* root 2004-08-05 18:29 11101 New_database.kexi bainm 2005-01-06 20:36 10240 projections.sxc bainm 2004-12-21 13:33 7597 testhtml.sxw bainm 2005-01-06 11:33 5529

The pipes and filters allow us to create an elegant piece of scripting. Now, instead of five individual commands, we have a single, flowing process.

Some useful filters

There are many Linux commands that are filters, in addition to awk , grep , and sort . Two filters to consider are tr (translate) and sed (stream edit). Both commands allow you to modify the stream — tr for simple changes and sed for the more complex. For example, you can use tr [a-z] [A-Z] to convert everything to uppercase, or sed s/"*"//g to remove the stars from the names of executable files.

Another filter to consider is tee , which enables you to split a stream between stdout and a file. For example:

This will create a file (file.lst) containing the result from ls -l and will display the number of files to the screen (or pass it on to another filter, if you require).

Creating your own filters

So far, we have learned how to use pipes and simple filters together. The next step is to learn how to build a filter for a specific job. The above example will send a list of all the files in the home directory. However, let’s assume that we’re interested only in files that are greater than 10,000 bytes in size. We need to add in a new filter:

ls -l ~ | grep -v "^d" | awk '' | only_big_files | sort -nr | awk '' | mail -s "File List" info@markbain-writer.tk >

The filter must first read the standard input. To do this, enclose any functionality within a “while read” loop. Any fields passed to the filter must be placed into variables:

while read SIZE FILE NAME DATE TIME do. done

Having read the standard input, we can now create the body of the filter. Here we simply check to see if the file is greater than 10,000. If it is, we send the data to the standard output. If not, we move onto the next line:

if [ $SIZE -gt 10000 ] then echo $SIZE $FILE $NAME $DATE $TIME fi

You could, of course, use the awk filter to do the same:

Final thoughts

I find pipes and filters invaluable. Their uses range from simple processes (such as ls -l | more ) through to the highly complex. Like so many things in Linux, you’ll wonder how you ever managed to live without them.

Источник

What are filters in Linux?

What are filters in Linux? Filters are programs that take plain text(either stored in a file or produced by another program) as standard input, transforms it into a meaningful format, and then returns it as standard output. Linux has a number of filters.

Which is example of filter in Linux? In UNIX/Linux, filters are the set of commands that take input from standard input stream i.e. stdin, perform some operations and write output to standard output stream i.e. stdout. The stdin and stdout can be managed as per preferences using redirection and pipes. Common filter commands are: grep, more, sort. 1.

What is pipes and filters in Linux? A pipe can pass the standard output of one operation to the standard input of another, but a filter can modify the stream. A filter takes the standard input, does something useful with it, and then returns it as a standard output. Linux has a large number of filters.

What is the use of filters in Unix? In Unix and Unix-like operating systems, a filter is a program that gets most of its data from its standard input (the main input stream) and writes its main results to its standard output (the main output stream).

What are filters in Linux? – Additional Questions

Why we use filter in Linux?

Linux Filter commands accept input data from stdin (standard input) and produce output on stdout (standard output). It transforms plain-text data into a meaningful way and can be used with pipes to perform higher operations.

What is the use of filter?

1 : a device or a mass of material (as sand or paper) with tiny openings through which a gas or liquid is passed to remove something The filter removes dust from the air. 2 : to remove by means of a filter Sand helps filter impurities from water.

How do you filter records in Unix?

1. Awk Command. Awk is a remarkable pattern scanning and processing language, it can be used to build useful filters in Linux.
2. Sed Command.
3. Grep, Egrep, Fgrep, Rgrep Commands.
4. head Command.
5. tail Command.
6. sort Command.
7. uniq Command.
8. fmt Command.

What is the function of the find filter command?

The FILTER function “filters” a range of data based on supplied criteria. The result is an array of matching values from the original range. In plain language, the FILTER function will extract matching records from a set of data by applying one or more logical tests.

What is the difference between pipes and filters?

A pipe or a filter? The pipe is just the | that connects the output of one program to the input of the next. A filter is any program which performs some operation on data received via stdin and outputs the “filtered” data via stdout.

What is filter and pipes?

Pipe and Filter is another architectural pattern, which has independent entities called filters (components) which perform transformations on data and process the input they receive, and pipes, which serve as connectors for the stream of data being transformed, each connected to the next component in the pipeline.

What is simple filters?

Simple filters provide a way to target a set of records in a list based on specified conditions.

What is pipe in shell?

The pipe character | is used to connect the output from one command to the input of another. > is used to redirect standard output to a file. Try it in the shell-lesson-data/exercise-data/proteins directory!

What is advantage of filter and pipe?

• Easy to understand the overall input and output. behavior of the system.
• Support reuse.
• Support deadlock analyses feature.
• Concurrent execution which improve performance.
• Easy to maintain and enhance; new filter can be.

What is filter software architecture?

A filter is an independent data stream transformer or stream transducers. It transforms the data of the input data stream, processes it, and writes the transformed data stream over a pipe for the next filter to process.

What layered styles?

The layered architecture style is one of the most common architectural styles. The idea behind Layered Architecture is that modules or components with similar functionalities are organized into horizontal layers. As a result, each layer performs a specific role within the application.

What is process control architecture?

In process control architecture, the flow of data comes from a set of variables which controls the execution of process. This architecture decomposes the entire system into subsystems or modules and connects them.

What is the DFD diagram?

A data flow diagram (DFD) is a graphical or visual representation using a standardized set of symbols and notations to describe a business’s operations through data movement. They are often elements of a formal methodology such as Structured Systems Analysis and Design Method (SSADM).

What is flow in architecture?

What is interior flow? Interior flow, also referred to as the circulation of a building, is the natural movement of people and air between rooms. The placement of hallways, corridors, doors and windows are critical to establishing a flow that allows a home to “breathe” successfully.

What is a local control system?

The local control unit collects all the quantities needed to determine the state of the local resources and generates the reference set point of the power to be injected into the grid.

What is local panel?

Local control panels are generally there to control the machine or equipment locally or remotely and installed in the close proximity of the controlled device. These panels can house more devices than the operator push buttons.

What is a local control station?

Local Control stations (LCS) or otherwise termed as Push Button Station (PBS) are an integral part of control gear for any heavy industrial scenario, they are used for controlling the machinery either locally by the front line engineer or remotely from the control room depending on the application of the machine or

Источник