File locking linux programming

Introduction to File Locking in Linux

The Kubernetes ecosystem is huge and quite complex, so it’s easy to forget about costs when trying out all of the exciting tools.

To avoid overspending on your Kubernetes cluster, definitely have a look at the free K8s cost monitoring tool from the automation platform CAST AI. You can view your costs in real time, allocate them, calculate burn rates for projects, spot anomalies or spikes, and get insightful reports you can share with your team.

Connect your cluster and start monitoring your K8s costs right away:

1. Overview

File locking is a mutual-exclusion mechanism to ensure a file can be read/written by multiple processes in a safe way.

In this tutorial, we’ll understand the interceding update problem in a multiple-processes system. Then, we’re going to introduce two types of locks in Linux.

Along the way, we’ll learn some file-locking-related commands through examples.

2. The Interceding Update Problem

The interceding update is a typical race condition problem in a concurrent system. Let’s see an example to understand the problem better.

Let’s say we have a balance.dat file storing the balance of an account, and it has an initial value of “100“. Our concurrent system has two processes to update the balance value:

1. Process A: reads the current value, subtracts 20 and saves the result back to the file.
2. Process B: reads the current value, adds 80 and writes the result back into the file.

Obviously, after the execution of two processes, we are expecting the file has value: 100-20+80=160.

However, an interceding update problem may occur in this situation:

1. Process A reads the file’s current value (100) and prepares to do further calculation.
2. Process B now reads the same file and gets the current balance (100).
3. Process A calculates 100-20 and saves the result 80 back to the file.
4. Process B doesn’t know the balance has been updated since its last read. So, it will still use the stale value 100 to calculate 100+80 and write the result 180 to the file.

As a result, we have 180 in the balance.dat file instead of the expected value 160.

3. File Locking in Linux

File locking is a mechanism to restrict access to a file among multiple processes. It allows only one process to access the file in a specific time, thus avoiding the interceding update problem.

We all know that rm -rf / is a very dangerous command in Linux. If we execute the command as the root user, all files in the running system will be deleted. This is because Linux usually doesn’t automatically lock open files. However, Linux supports two kinds of file locks: advisory locks and mandatory locks.

We’ll introduce both lock types shortly, but this article will primarily focus on advisory locking.

3.1. Advisory Locking

Advisory locking is not an enforced locking scheme. It will work only if the participating processes are cooperating by explicitly acquiring locks. Otherwise, advisory locks will be ignored if a process is not aware of locks at all.

An example may help us to understand the cooperative locking scheme easier. Let’s review our previous balance example.

1. First, we assume that the file balance.dat still contains the initial value of “100“.
2. Process A acquires an exclusive lock on the balance.dat file, then opens and reads the file to get the current value: 100.

We must understand that the advisory lock was not set by the operating system or file system. Therefore, even if process A locks the file, process B is still free to read, write, or even delete the file via system calls.

If process B executes file operations without trying to acquire a lock, we say process B is not cooperating with process A.

But now, let’s have a look at how the lock will work for cooperating processes:

1. Process B tries to acquire a lock on the balance.dat file before reading the file (cooperating with process A).
2. Since process A has locked the file, process B has to wait for process A to release the lock.
3. Process A calculates 100-20 and writes 80 back to the file.
4. Process A releases the lock.
5. Process B now acquires a lock and reads the file, and it gets the updated value: 80.
6. Process B starts its logic and writes the result 160 (80+80) back to the file.
7. Process B releases the lock so that other cooperating processes can read and write to the file.

We’ll see how the example is implemented with the flock command in a later section.

3.2. Mandatory Locking

Before we start looking at mandatory file locking, we should keep in mind that “Linux implementation of mandatory locking is unreliable“.

Unlike advisory locking, mandatory locking doesn’t require any cooperation between the participating processes. Once a mandatory lock is activated on a file, the operating system prevents other processes from reading or writing the file.

To enable mandatory file locking in Linux, two requirements must be satisfied:

1. We must mountthe file system with the mand option (mount -o mand FILESYSTEM MOUNT_POINT).
2. We must turn on the set-group-ID bit and turn off the group-execute bit for the files we are about to lock (chmod g+s,g-x FILE).

4. Inspect All Locks in a System

In this section, let’s have a look at two ways to inspect the currently acquired locks in a running system.

4.1. The lslocks Command

The lslocks command is a member of the util-linux package and available on all Linux distributions. It can list all currently held file locks in our system.

Let’s see an example output:

$ lslocks COMMAND PID TYPE SIZE MODE M START END PATH lvmetad 298 POSIX 4B WRITE 0 0 0 /run/lvmetad.pid containerd 665 FLOCK 128K WRITE 0 0 0 /var/lib/docker/. chromium 184029 POSIX 9.4M WRITE 0 1073741824 1073742335 /home/kent/.config/chromium/Default/History nextcloud 961 POSIX 32K READ 0 128 128 /home/kent/Nextcloud/._sync_0e131dbf228b.db-shm dockerd 630 FLOCK 16K WRITE 0 0 0 /var/lib/docker/buildkit/snapshots.db dropbox 369159 FLOCK 10M WRITE 0 0 0 /home/kent/.dropbox/logs/1/1-4ede-5e20dd8d.tmp . 

In the above list, we can see all the currently locked files in the system. We can also see detailed information of each lock, such as the lock type, and which process holds the lock.

4.2. /proc/locks

/proc/locks is not a command. Instead, it is a file in the procfs virtual file system. The file holds all current file locks. The lslocks command relies on this file to generate the list, too.

To get the information of /proc/locks, we execute “cat /proc/locks“:

$ cat /proc/locks 1: FLOCK ADVISORY WRITE 369159 08:12:22417368 0 EOF 2: POSIX ADVISORY WRITE 321130 00:2e:30761 0 EOF 3: POSIX ADVISORY WRITE 184029 08:12:21760394 0 EOF 4: POSIX ADVISORY WRITE 184029 08:12:21633968 1073741824 1073742335 5: POSIX ADVISORY WRITE 184029 08:12:21760401 0 EOF 6: POSIX ADVISORY WRITE 184029 08:12:21891515 0 EOF 7: POSIX ADVISORY WRITE 184029 08:12:21633928 0 EOF . 

Let’s pick the first row to understand how the lock information is organized in the /proc/locks file system:

1: FLOCK ADVISORY WRITE 369159 08:12:22417368 0 EOF -1- --2-- ---3--- --4-- ---5-- -------6------ -7- -8- 

1. The first column is a sequence number.
2. The second field indicates the class of the lock used, such as FLOCK (from flock system call) or POSIX (from the lockf, fcntl system call).
3. This column is for the type of lock. It can have two values: ADVISORY or MANDATORY.
4. The fourth field reveals if the lock is a WRITE or READ lock.
5. Then we have the ID of the process holding the lock.
6. This field contains a colon-separated-values string, showing the id of the locked file in the format of “major-device:minor-device:inode“.
7. This column, together with the last one, shows the start and end of the locked region of the file being locked. In this example row, the entire file is locked.

5. Introduction to flock Command

The flock command is also provided by the util-linux package. This utility allows us to manage advisory file locks in shell scripts or on the command line.

The basic usage syntax is:

flock FILE_TO_LOCK COMMAND

Next, let’s demonstrate our balance update example with the flock command.

In addition to a balance.dat text file containing the current balance value, we still need two processes, A and B, to update the balance in the file.

We first create a simple shell script update_balance.sh to handle balance update logic for both processes:

#!/bin/bash file="balance.dat" value=$(cat $file) echo "Read current balance:$value" #sleep 10 seconds to simulate business calculation progress=10 while [[ $progress -lt 101 ]]; do echo -n -e "\033[77DCalculating new balance..$progress%" sleep 1 progress=$((10+progress)) done echo "" value=$((value+$1)) echo "Write new balance ($value) back to $file." echo $value > "$file" echo "Done."

We create a simple shell script a.sh to simulate process A:

#!/bin/bash #----------------------------------------- # process A: lock the file and subtract 20 # from the current balance #----------------------------------------- flock --verbose balance.dat ./update_balance.sh '-20'

Now let’s start process A to test:

$ ./a.sh flock: getting lock took 0.000002 seconds flock: executing ./update_balance.sh Read current balance:100 Calculating new balance..100% Write new balance (80) back to balance.dat. Done.

Through the output, we can see the flock command first acquired a lock on the file balance.dat, then the update_balance.sh script read and updated the file.

During its run, we can check the lock information via the lslocks command:

$ lslocks | grep 'balance' flock 825712 FLOCK 4B WRITE 0 0 0 /tmp/test/balance.dat

The output shows that the flock command is holding a WRITE lock on the entire file /tmp/test/balance.dat.

5.1. Demonstration of flock With Non-Cooperative Processes

We’ve learned that the advisory locks work only if participating processes are cooperating. Let’s reset the balance to 100 and check what will happen if we acquire an advisory lock on the file for process A but start the process B in a non-cooperative way.

Now let’s create a simple shell script b_non-cooperative.sh:

#!/bin/bash #---------------------------------------- # process B: add 80 to the current balance in a # non-cooperative way #---------------------------------------- ./update_balance.sh '80'

We see that process B calls update_balance.sh without attempting to acquire a lock on the balance data file.

Let’s demonstrate this scenario in a GIF animation:

We see that the advisory lock acquired by process A is ignored if process B starts without cooperating with process A.

Therefore, we have 180, instead of 160, in the balance.dat.

5.2. Demonstration of flock With Cooperative Processes

Finally, let’s create another cooperative process B, b.sh, and see how the advisory lock works:

#!/bin/bash #---------------------------------------- # process B: add 80 to the current balance # in a cooperative way #---------------------------------------- flock --verbose balance.dat ./update_balance.sh '80'

Again, we show the demonstration in a GIF animation:

In the demo, we made two processes to cooperate.

We noticed that when process B attempted to acquire a lock on the balance.dat file, it waited for process A to release the lock. Thus, the advisory locking worked, and we got the expected result, 160, in the balance data file.

6. Conclusion

In this article, we started by understanding the interceding update problem. Then we discussed different types of file locking schemes in Linux systems.

We also learned the lslocks command to check locks in a system and the flock utility to implement advisory locking.

Finally, we saw two demonstrations. One helped us to understand the relationship between advisory locking and process cooperation, while the other showed us how advisory locking works.

Источник