How do I atomically create a locked file in Linux?
Scenario: I have many processes running that need to fetch files over the net. If the file is already downloaded, I want it cached on disk. If another process is downloading the file, block until it is finished downloading. I’ve been trying to find the easiest way to do to this. The obvious way is to:
create file w/ an exclusive lock active on it only if it doesn't exist (O_CREAT | O_EXCL) if file exists already: open file and acquire exclusive lock else: download to newly created file release lock This system accomplishes the above goals with (seemingly) no race conditions Unfortunately, I couldn’t find documentation on how to use open(), etc. to create a file that is locked in Linux. If I split the create step into:
open w/ O_CREAT | O_EXCL flock a race condition now exists between the create and lock (non-creating process acquires the lock before the creator does). I realize I could use an external lock file per file (e.g. filename + ‘.lock), which I acquire before attempting to create filename, but this feels.. inelegant (and I need to now worry about how to files that actually have a .lock suffix!) Is there anyway to atomically create and lock it (as Windows offers) or is the external lockfile method pretty much what is standard/required?
3 Answers 3
The race exists anyway. If the file may or may not exist then you have to test for its existence before trying to lock it. But if the file is your mutex, then you can’t possibly do that and the space between «if file exists already» (false) and «download to newly created file» is unconstrained. Another process could come by and create the file and start downloading before your download begins, and you would clobber it.
Basically don’t use fcntl locks here, use the existence of the file itself. open() with O_CREAT and O_EXCL will fail if the file already exists, telling you that someone else got there first.
I’m not sure how the race would exist if I could atomically create a locked file only if the file does not exist. (condition is also atomic). O_CREAT | O_EXCL is an atomic create if file doesn’t exist; I just want to do that with a lock. Another process can’t start downloading as it will detect the file exists. Finally, I can’t use open alone as I must block until the download is done; in my solution, I rely on the (exclusive read/write) lock.
You’re right, atomically creating and locking a file would fix your problem. But that’s not the way fcntl locks work, sorry. If you want to use a fcntl lock, the file your are locking has to exist before any synchronization action is taken by your program. Find a way to move the creation outside that if() in your pseudocode, or else work up another synchronization protocol that doesn’t use the download file as the mutex.
Got it, so the answer is that Linux doesn’t offer Windows style atomic locks with opening. I’ll stick with the lockfile method.
Why don’t you use a lockfile utility?
Examples
Suppose you want to make sure that access to the file «important» is serialised, i.e., no more than one program or shell script should be allowed to access it. For simplicity’s sake, let’s suppose that it is a shell script. In this case you could solve it like this:
. lockfile important.lock . access_"important"_to_your_hearts_content . rm -f important.lock . I’m struggling with how to solve a similar problem at the moment, which is what brought me to your question. As I see it, the essence is:
int fd = open(path, O_CREAT|O_RDWR|O_EXCL, mode); if (fd == -1) < /* File already exists. */ the_file_already_exists(fd); >else < /* I just now created the file. Now I'll lock it. */ /* But first I'll deliberately create a race condition!! */ deliberately_fork_another_process_that_handles_file(path); int code = flock(fd,LOCK_EX); if (code < 0) < perror("flock"); exit(1); >/* I now have the exclusive lock. I can write to the file at will -- or CAN I?? See below. */ write_to_the_file_at_will(fd); > Obviously in real life I would never deliberately create that race condition, but its equivalent could certainly happen by accident in a real system. That other process might, for example, open the file for reading, obtain a shared lock on it, and read the file. It would see an empty file. That could mean that a write operation is in progress, but it might mean that the file is simply empty and that’s the correct and final answer.
If empty files are not allowed, the reader could simply behave exactly the way it would behave if the file was missing. After all, if the reader had started a millisecond earlier it would have failed to open the file anyway. In this case the reader needs to check if the file is empty after it opens it.
If empty files ARE allowed, then you’re in a bit of a quandary and I have no ready answer for that.
The problem I have is that when a file is first created, I want to write some sort of default value into it, because I want to «auto-initialize» a fresh system without having to pre-create every possible file it might need. That other process handling the file might itself have already initialized it! For all I know, three other processes might have also run in the meantime and altered the value. In that case I certainly do not want to «write to the file at will» after obtaining the exclusive lock, because I will clobber all those changes.
I suppose the answer is for my code above to ensure that the file is empty before writing to it. If it is NOT empty, then the code should behave exactly as if the file already existed: i.e., it should call:
the_file_already_exists(fd); Perhaps the bottom line to all of this discussion is that every process which handles the file in any way should check to see if it is empty and behave accordingly. Again though, if empty files ARE allowed, then I can’t yet think of any guaranteed solution. None of this would be necessary if there were some way to create the file and lock it as a single atomic sequence, but I don’t think there is any way to do that.
How to Lock a Text File in Linux Using flock Command
Before we explore the techniques/approaches of locking a text file under a Linux operating system environment, we should first understand the logic behind Linux’s file locking mechanism.
Linux’s file locking mechanism restricts/controls file access among multiple processes. When a text file is successfully locked, only one process can access it on a specific time schedule.
Before proceeding with this article, please understand that file locking is very much different from file encryption or file access control where a passphrase or password is needed to control user access to your files.
When a file is locked under a Linux operating system environment, a mutual exclusion event is created where only a single process can access it at a time.
Problem Statement
The Linux operating system will automatically block any write action attempts directed to an open file being written on by another process. However, what if you want to revoke the read and/or write permissions already invoked by the first process that has your file open and in write mode? Is there a workaround to this problem?
Moreover, we might also want to lock our file so that no other process interferes or attempt to disrupt the write mode status already initiated. This article will walk us through a viable solution to lock a text file in Linux.
Sample Reference File
For this tutorial to be more engaging and worthwhile, we should have some reference text files. We will concentrate on the text files inside the following directory:
For instance, let us open the file output.txt and start writing on it.
While this file is still open, let us try to open it from another command line tab.
You will get the following response:
The above screen capture relays the PID (133110) of the process of working on the text file with the option of opening the file while it is still in that process’s write mode.
By keying in Y from our keyboard, we will have opened this text file and handed it over to a different process with exclusive write mode access.
Therefore, a file modification by user 1 will lead to the following prompt on user 2 while attempting to save the file.
This instance is a perfect representation of two different users on the network working on a single file.
Lock a Text File Using Linux’s flock Utility
To solve this issue, we need the aid of Linux’s flock utility. Since the util-linux package hosts flock command, it should be available in almost all Linux OS distributions. This command manages specific file/directory locks via the Linux command line environment.
To lock a text file in Linux, we will reference the following syntax:
$ flock -x PATH_TO_FILE_TO_LOCK -c COMMAND
The -x option is for obtaining a write lock to the targeted file. Let us attempt to lock the sample output.txt text file. The -c option will enable us to pass a single Linux supported command e.g. cat command.
$ flock -x /home/dnyce/LinuxShellTips_Files/output.txt -c cat
The terminal instance above will remain active to signify that the text file has been locked.
If we open another terminal (while this terminal instance is still running) and execute another flock command on this same file, we should be denied access to it.
$ flock -w .007 /home/dnyce/LinuxShellTips_Files/output.txt -c echo; /bin/echo $? 1
The -w option is used to relay the wait time of .007 seconds before a lock is placed on the text file. We then execute echo $? to output the exit status of this command.
An exit status of 0 implies that the command was executed successfully and an exit status of 1 implies the command could not be executed due to an error. In this case, the text file is under lock by another process.
To further confirm that the text file is under lock, we can use the lslocks commands to list all active local system locks.
As you can see, our file is present.
Unlocking a Text File in Linux
Canceling the initial flock command (Ctrl+c) or closing the text file should release the lock making it possible to successfully run the following command on the secondary terminal.
$ flock -w .007 /home/dnyce/LinuxShellTips_Files/output.txt -c echo; /bin/echo $?
The exit status of 0 implies that the lock is no longer applicable to the text file.
We have learned how to lock a text file in Linux so that only one process can use it (write mode) at a time. The implementation of this article is particularly useful when different users on a network are accessing a single file.
More on flock command can be found on its man page.