Dev shm on linux

How and when to use /dev/shm for efficiency?

How is /dev/shm more efficient than writing the file on the regular file system? As far as I know, /dev/shm is also a space on the HDD so the read/write speeds are the same. My problem is, I have a 96GB file and only 64GB RAM (+ 64GB swap). Then, multiple threads from the same process need to read small random chunks of the file (about 1.5MB). Is /dev/shm a good use case for this?
Will it be faster than opening the file in read-only mode from /home and then passing over to the threads to do the reading the required random chunks?

/dev/shm is not on the hard disk. It’s a virtual filesystem implemented in memory, that’s why it’s faster.

@Barmar right, just realised that. Then makes sense to use it, especially since it can be shared from multiple processes so it won’t take 96GB per process that wants to use it.

1 Answer 1

You don’t use /dev/shm . It exists so that the POSIX C library can provide shared memory support via the POSIX API. Not so you can poke at stuff in there.

If you want an in-memory filesystem of your very own, you can mount one wherever you want it.

mount -t tmpfs tmpfs /mnt/tmp , for example.

A Linux tmpfs is a temporary filesystem that only exists in RAM. It is implemented by having a file cache without any disk storage behind it. It will write its contents into the swap file under memory pressure. If you didn’t want the swapfile you can use a ramfs .

I don’t know where you got the idea of using /dev/shm for efficiency in reading files, because that isn’t what it does at all.

Maybe you were thinking of using memory mapping, via the mmap system call?

Источник

What is /dev/shm and What is it Used For?

/dev/shm, also known as the “shared memory file system,” is a special file system in Linux that is used for storing temporary files in memory. It is a temporary storage area in RAM that different processes can use to share data quickly and efficiently.

History of /dev/shm

The concept of shared memory has been around for a long time, dating back to the early days of computer science.

In the early days of computing, shared memory was used to allow different processes to communicate with each other and exchange data without the need for complex interprocess communication (IPC) mechanisms.

With the advent of Unix and Linux operating systems, shared memory became a more integral part of the operating system itself. In the early days of Unix, shared memory was implemented using a system called “shmget,” which allowed processes to create, attach, and detach shared memory segments.

However, with the growth of the Linux operating system, shared memory became more important and was implemented as a special file system called “/dev/shm.”

This allowed shared memory to be accessed and used by processes in a more user-friendly manner, similar to how other files and directories are accessed in a file system.

How /dev/shm Works?

/dev/shm works by creating a temporary storage area in RAM that is accessible to all processes on the system. When a process wants to create a shared memory segment, it can do so by calling the “shmget” system call and specifying the size and permissions for the shared memory segment.

Once the shared memory segment has been created, the process can then attach to it using the “shmat” system call. This allows the process to access the shared memory segment and read or write data to it as needed.

When the process is finished using the shared memory segment, it can detach from it using the “shmdt” system call. This frees up the memory for other processes and allows the operating system to reclaim the memory if needed.

What are the Benefits of Using /dev/shm?

/dev/shm has several benefits that make it a helpful tool for developers and system administrators.

Some of the main benefits include:

Fast Data Access

One of the main benefits of using /dev/shm is its fast data access. Since shared memory segments are stored in RAM, they can be accessed much faster than data stored on a hard drive or another storage device.

This makes /dev/shm ideal for storing temporary data that needs to be accessed quickly, such as buffers for I/O operations or data that is frequently accessed by multiple processes.

Efficient Data Sharing

/dev/shm also allows for efficient data sharing between processes. Since the shared memory segments are stored in a central location in RAM, multiple processes can simultaneously access and modify the data in the segments.

This eliminates the need for processes to communicate with each other using slower IPC mechanisms, such as pipes or sockets, which can be costly in terms of performance.

Reduced Disk I/O

Another benefit of using /dev/shm is the reduction in disk I/O it can provide. Since shared memory segments are stored in RAM, they are not written to disk like regular files.

This can help reduce the load on the disk and improve overall system performance.

Easy to Use

/dev/shm is also relatively easy to use, as it is accessed and used similarly to regular files and directories in a file system. This means that developers and system administrators can use familiar tools, such as cp , mv , and rm , to manage shared memory segments.

What are the Limitations of /dev/shm?

While /dev/shm has many benefits, it is essential to note that it is not a replacement for a traditional file system.

There are several limitations to using /dev/shm, including:

Limited Storage Capacity

One of the main limitations of /dev/shm is that it is limited by the amount of available RAM on the system. This means that shared memory segments cannot be larger than the available RAM on the system.

This can be a problem if a process needs to create a large shared memory segment without enough RAM available.

Volatility

Another limitation of /dev/shm is that it is volatile, meaning that data stored in shared memory segments is lost when the system is shut down or restarted.

This means that shared memory segments should not be used to store important or long-term data that needs to be preserved.

Security Concerns

/dev/shm can also pose security concerns if not used properly. Since shared memory segments are accessible to all processes on the system, it is important to properly set permissions on the segments to prevent unauthorized access or modification.

What are Examples of /dev/shm in Use?

/dev/shm is commonly used in various applications and scenarios.

I/O Buffering

One everyday use of /dev/shm is for I/O buffering. When a process needs to read or write data to a file, it can use a shared memory segment as a buffer to improve performance.

This allows the process to read or write data to the shared memory segment faster than reading or writing directly to the file on disk. Once the data has been reported to the shared memory segment, it can be written to the file on disk more efficiently.

Interprocess Communication

/dev/shm is also frequently used for interprocess communication (IPC). When two or more processes need to communicate with each other and exchange data, they can use a shared memory segment as a medium for communication.

This eliminates the need for slower IPC mechanisms, such as pipes or sockets, and allows the processes to communicate and exchange data more efficiently.

Database Caching

Another common use of /dev/shm is for database caching. When a database needs to access data from a disk, it can be slow and resource-intensive. To improve performance, databases can use a shared memory segment as a cache to store frequently accessed data in RAM.

This allows the database to access the data faster, as it does not need to read from the disk every time it is needed.

Application Cache

/dev/shm is also often used by applications as a cache to store frequently accessed data in RAM. This can improve the application’s performance by allowing it to access the data faster rather than having to read it from a slower storage device every time it is needed.

Conclusion

/dev/shm is a powerful tool that allows processes to share data quickly and efficiently using shared memory segments in RAM.

While it has several limitations, such as limited storage capacity and volatility, it is a helpful tool for various applications and scenarios, including I/O buffering, interprocess communication, database caching, and application caching.

By understanding how /dev/shm works and its benefits and limitations, developers and system administrators can effectively utilize shared memory to improve the performance of their systems and applications.

It is important to remember that /dev/shm should not be used to store essential or long-term data, as it is volatile and will be lost when the system is shut down or restarted. Instead, it should be used for temporary data that needs to be accessed quickly and efficiently.

Источник