Number of file descriptors linux

Linux Interview Questions : Open Files / Open File Descriptors

An open file may be a regular file, a directory, a block special file, a character special file, an executing text reference, a library, a stream or a network file.

What is file descriptor?

A file descriptor is a data structure used by a program to get a handle on a file. The most commonly known are:

0 for standard in 1 for standard out 2 for standard error

Does the number of open files equal the number of open file descriptors?

There is a difference in the number of current open files and the number of current file descriptors/handlers. Even though a file is open, it might not have a file descriptor associated with it such as current working directories, memory mapped files and executable text files.

The ‘lsof’ command will provide the number of open files associated with a process. For Example, for the process with PID 4448:

# lsof | grep -w 2374 pickup 2374 postfix cwd DIR 202,1 4096 8945851 /var/spool/postfix pickup 2374 postfix rtd DIR 202,1 4096 128 / pickup 2374 postfix txt REG 202,1 285112 8945807 /usr/libexec/postfix/pickup pickup 2374 postfix mem REG 202,1 62184 463326 /usr/lib64/libnss_files-2.17.so pickup 2374 postfix mem REG 202,1 155744 758332 /usr/lib64/libselinux.so.1 pickup 2374 postfix mem REG 202,1 15688 125778 /usr/lib64/libkeyutils.so.1.5 pickup 2374 postfix mem REG 202,1 58728 127331 /usr/lib64/libkrb5support.so.0.1 pickup 2374 postfix mem REG 202,1 11384 37284 /usr/lib64/libfreebl3.so .

# lsof | grep -w 2374 | wc -l 51

There are total 51 open files associated with PID 2374 (postfix).

To determine the number of open file descriptors associated with a process use the following. For the same postfix process, 2374:

# ls -l /proc/2374/fd total 0 lrwx------. 1 root root 64 Nov 18 18:06 0 -> /dev/null lrwx------. 1 root root 64 Nov 18 18:06 1 -> /dev/null l-wx------. 1 root root 64 Nov 18 18:06 10 -> pipe:[28867] lrwx------. 1 root root 64 Nov 18 18:06 2 -> /dev/null lr-x------. 1 root root 64 Nov 18 18:06 3 -> pipe:[18701] l-wx------. 1 root root 64 Nov 18 18:06 4 -> pipe:[18701] lrwx------. 1 root root 64 Nov 18 18:06 5 -> socket:[18631] lrwx------. 1 root root 64 Nov 18 18:06 6 -> socket:[18629] lrwx------. 1 root root 64 Nov 18 18:06 7 -> socket:[28851] lrwx------. 1 root root 64 Nov 18 18:06 8 -> anon_inode:[eventpoll] lr-x------. 1 root root 64 Nov 18 18:06 9 -> pipe:[28867] l-wx------. 1 root root 64 Nov 18 18:06 92 -> pipe:[18702]

There are 13 file descriptors associated with PID 2374 (postfix). We can see there is a difference between the number of open files and the number of file descriptors per process. postfix has 51 open files but only 13 file descriptors associated with it. Some of the open files do not have file descriptors associated with them; e.g., libraries, current working directories.

How to get the maximum number of file descriptors?

In Linux, maximum number of file descriptors limit can be read from /proc file system. To get the current limit on the number of file descriptors for the whole system use the following command.

# cat /proc/sys/fs/file-max 180451

How to compute the number of file descriptors currently being used?

Use the following command.

# cat /proc/sys/fs/file-nr 1344 0 180451 | | |_ Max no. of file descriptors allowed on the system | | | |__ Total free allocated file descriptors | |__ Total allocated file descriptors

To compute the number of file descriptors currently being used:

How to get the per-process limits on the file descriptors?

Use ‘ulimit -a‘ to get the per-process limits.

# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 7132 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 4096 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 7132 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

How does one find the number of open file descriptors being used by any given process?

Use the following command (Substitute your PID)

What are some other useful parameters?

NR_OPEN = Maximum number of open files per process NR_FILE = Total number of files that can be open in the system at any time FILE-MAX = Kernel parameter refers to maximum number of file descriptors allowed per system FILE-NR = Refers to the number of current file descriptors used at any moment. LSOF = Gives the number of open files.

Some more articles you might also be interested in …

Источник

How to get number of opened file descriptors for user

I know about lsof and ls /proc/*/fd but none of them are atomic AFAIK. Because in the latter case I would need to get all pids for user and then filter by them and by that time some of the file descriptors could be closed. Maybe there is some system call for that or something, because obviously OS tracks that number as it would refuse to create FD if max limit for user is exhausted.

Are you sure that it would refuse to create FD if max limit for user is exhausted ? The setrlimit/getrlimit system calls work a per-process base. diskquota works on a per filesystem base. AFAIK, there is no API that works on a per user base.

Well, there is a limit for every user. If it wouldn’t refuse to create file when it’s exhausted then this limit is useless, right?

either lsof (with non atomic constraint) or read kernel sources and build your own utility. A good question anyway.

@user1685095 There is a limit for each process of a specified user not for each user. If a user has an hard limit of RLIMIT_NOFILE set to 100, she’ll can have two processes with 99 open files (198 in total).

1 Answer 1

I haven’t made an intensive search, but I don’t think what you’re looking for exists on Linux. Opening a file descriptor doesn’t take any global lock, only a per-process lock, so on a multicore machine whatever you’d be using to count the number of open file descriptors could be running literally at the same time that other threads is opening or closing files on other cores.

Linux doesn’t have a global limit on the total number of open files. There’s no explicit per-user limit either. There’s a per-user limit on processes, and a per-process limit on file descriptor numbers, which indirectly imposes a limit on open files per user, but that isn’t explicitly tracked.

Exploring /proc (which is what lsof does under the hood) is as good as it gets. /proc is the Linux API to get information about processes.

gerardnico.com/wiki/linux/limits.conf you can see there that there is a limit for open file descriptors. Which is exactly what I’m talking about.

@user1685095 As several of us have told you already, this is a per process limit, not per user. limits.conf defines different values for each user, but what that means is that each process run by the user has this limit, not that the limit is for the total number of files open by a process run by the user.

Источник

Counting open files per process

I’m working on an application that monitors the processes’ resources and gives a periodic report in Linux, but I faced a problem in extracting the open files count per process. This takes quite a while if I take all of the files and group them according to their PID and count them. How can I take the open files count for each process in Linux?

5 Answers 5

Have a look at the /proc/ file system:

To do this for all processes, use this:

cd /proc for pid in 2* do echo "PID = $pid with $(ls /proc/$pid/fd/ | wc -l) file descriptors" done 

As a one-liner (filter by appending | grep -v «0 FDs» ):

for pid in /proc/4*; do printf "PID %6d has %4d FDs\n" $(basename $pid) $(ls $pid/fd | wc -l); done 

As a one-liner including the command name, sorted by file descriptor count in descending order (limit the results by appending | head -10 ):

for pid in /proc/8*; do p=$(basename $pid); printf "%4d FDs for PID %6d; command=%s\n" $(ls $pid/fd | wc -l) $p "$(ps -p $p -o comm=)"; done | sort -nr 

Credit to @Boban for this addendum:

You can pipe the output of the script above into the following script to see the ten processes (and their names) which have the most file descriptors open:

 . done | sort -rn -k5 | head | while read -r _ _ pid _ fdcount _ do command=$(ps -o cmd -p "$pid" -hc) printf "pid = %5d with %4d fds: %s\n" "$pid" "$fdcount" "$command" done 

Here’s another approach to list the top-ten processes with the most open fds, probably less readable, so I don’t put it in front:

find /proc -maxdepth 1 -type d -name '3*' \ -exec bash -c "ls <>/fd/ | wc -l | tr '\n' ' '" \; \ -printf "fds (PID = %P), command: " \ -exec bash -c "tr '\0' ' ' < <>/cmdline" \; \ -exec echo \; | sort -rn | head 

Of course, you will need to have root permissions to do that for many of the processes. Their file descriptors are kind of private, you know 😉

/proc/$pid/fd lists descriptor files, that is slightly different of «open files» as we can have memory map and other unusual file objects.

This extends the answer and turns pids to command names: for pid in 2*; do echo «PID = $pid with $(ls /proc/$pid/fd/ 2>/dev/null | wc -l) file descriptors»; done | sort -rn -k5 | head | while read -r line; do pid= echo $line | awk ‘‘ ; command= ps -o cmd -p $pid -hc ; echo $line | sed -s «s/PID = \(.*\) with \(.*\)/Command $command (PID = \1) with \2/g»; done

Yeah, well. Instead of parsing the original output and then call ps again for each process to find out its command, it might make more sense to use /proc/$pid/cmdline in the first loop. While technically it is still possible for a process to disappear between the evaluating of 4* and the scanning of its disc, this is less likely.

Executing command=$(ps -o cmd -p «$pid» -hc) gave me Warning: bad syntax, perhaps a bogus ‘-‘ . It worked running as command=$(ps -o cmd -p «$pid» hc) .

ps aux | sed 1d | awk '' | xargs -I <> bash -c <> 

For Fedora, it gives: lsof: WARNING: can’t stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs Output information may be incomplete. lsof: no pwd entry for UID 65535

I used this to find top filehandler-consuming processes for a given user (username) where dont have lsof or root access:

for pid in `ps -o pid -u username` ; do echo "$(ls /proc/$pid/fd/ 2>/dev/null | wc -l ) for PID: $pid" ; done | sort -n | tail 

How can I take the open files count for each process in Linux?

if you’re running it from root (e.g. prefixing the command with sudo -E env PATH=$PATH ), otherwise it’ll only return file descriptor counts per process whose /proc//fd you may list. This will give you a big JSON document/tree whose nodes look something like:

The content of fd dictionary is counts per file descriptor type. The most interesting ones are probably these (see procfile.Fd description or man fstat for more details):

I’m the author of Procpath, which is a tool that provides a nicer interface to procfs for process analysis. You can record a process tree’s procfs stats (in a SQLite database) and plot any of them later. For instance this is how my Firefox’s process tree (root PID 2468) looks like with regards to open file descriptor count (sum of all types):

procpath --logging-level ERROR record -f stat,fd -i 1 -d ff_fd.sqlite \ '$..children[?(@.stat.pid == 2468)]' # Ctrl+C procpath plot -q fd -d ff_fd.sqlite -f ff_df.svg 

If I’m interested in only a particular type of open file descriptors (say, sockets) I can plot it like this:

procpath plot --custom-value-expr fd_sock -d ff_fd.sqlite -f ff_df.svg 

Источник