Ram file in linux

creating a temporary file in memory and using it as input file of a command

There is a command pdflatex, which I want to use in my bash script. It takes as input a filename on which content it will work. Now I have an algorithms that looks as follows:

for stuff in list of stuff; do echo "$" > /tmp/tmpStuff pdflatex /tmp/tmpStuff done 

Now this works as expected. But I was thinking I could speed that up by doing less disk I/O(as > redirection writes to a file). I wish I could write something like echo «$stuff» | pdflatex /tmp/tmpStuff but pdflatex uses a file and not stdin as its input. Is there any way of keeping «$stuff» in memory and passing it to pdflatex as a sort of file? TLDR: I would be happy if I could create a temporary file which could be named and be in memory.

4 Answers 4

You can use process substitution for this:

From the Bash Reference Manual:

3.5.6 Process Substitution

Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of naming open files. It takes the form of

The process list is run with its input or output connected to a FIFO or some file in /dev/fd. The name of this file is passed as an argument to the current command as the result of the expansion. If the >(list) form is used, writing to the file will provide input for list. If the and the left parenthesis, otherwise the construct would be interpreted as a redirection.

And I also wonder if a here-string would make it as well:

Thank you for your answer, is there any way of naming the file resulting file «temporary» file coming from <(echo "$stuff") ? I'm wondering that because it seems to cause errors in pdflatex.

@Simonlbc You can check creating a named pipe as explained in Pseudo files for temporary data. However, it seems better to use process substitution.

Thank you for your help. It seems echo «$stuff» | pdflatex -jobname outputPdfName seems to work after all. I tried pdflatex

None of the presented solutions is guaranteed to work on all possible commands.

Consider myCommand below (instead of pdflatex ):

#!/bin/bash # myCommand test -f "$1" || < echo "error: Please provide a file"; exit 1; >echo "The file content is:"$(cat $1) 

Trying the existing solutions:

In my Ubuntu I use the following approach (utilizing /dev/shm in-memory file storage):

ramtmp="$(mktemp -p /dev/shm/)" echo "abcdef" > $ramtmp ./myCommand $ramtmp # The file content is:abcdef 

So in your case it would be:

ramtmp="$(mktemp -p /dev/shm/)" for stuff in list of stuff; do echo "$" > $ramtmp pdflatex $ramtmp done 

In case pdflatex execution is non-blocking add the first line inside the loop.

Note that, I’m not sure how many distributions currently support it. See Wikipedia article:

Linux distributions based on the 2.6 kernel and later offer /dev/shm as shared memory in the form of a RAM disk, more specifically as a world-writable directory (a directory in which every user of the system can create files) that is stored in memory. Both the RedHat and Debian based distributions include it by default. Support for this type of RAM disk is completely optional within the kernel configuration file.[5]

Источник

Файловая система в оперативной памяти — как пользоваться tmpfs

Файловая система tmpfs может найти повседневное применение в вашей деятельности, поскольку она невероятно быстрая и может помочь снизить нагрузку на ваше постоянное хранилище (особенно актуально тем, у кого Linux установлен на флешку или карту памяти).

tmpfs — это виртуальная файловая система, располагающаяся в оперативной памяти.

Средство tmpfs позволяет создавать файловые системы, содержимое которых находится в виртуальной памяти. Поскольку файлы в таких файловых системах обычно находятся в ОЗУ, доступ к файлам осуществляется очень быстро.

Файловая система создаётся автоматически при монтировании файловой системы с типом tmpfs с помощью следующей команды:

sudo mount -t tmpfs -o size=10M tmpfs /mnt/mytmpfs

Файловая система tmpfs имеет следующие свойства:

• Файловая система может использовать пространство подкачки, когда этого требует физическая нагрузка на память.
• Файловая система потребляет столько физической памяти и пространства подкачки, сколько требуется для хранения текущего содержимого файловой системы.
• Во время операции повторного монтирования (mount -o remount) размер файловой системы может быть изменён (без потери существующего содержимого файловой системы).

Если файловая система tmpfs размонтирована, её содержимое теряется (удаляется).

Вы можете скопировать в tmpfs файлы для максимально быстрого доступа. Это могут быть файлы баз данных или веб-сервера.

Ещё одна цель использования — снизить износ постоянного хранилища. Это не особенно актуально для жёсткого диска или твердотельного диска — современные модели при любом типе домашнего использования переживут нас. Но это может быть актуально, если система установлена на карту памяти. Вы можете разместить в оперативную память приложение, которое постоянно использует хранилище (часто обращается к файлам или непрерывно сохраняет файлы), тем самым ускорится работа этого приложения, а также всей системы за счёт снижения нагрузки на карту памяти.

Ещё одна возможная причина использование — незаметность, при работе в tmpfs всё будет происходить в оперативной памяти, а на постоянных хранилищах не останется никаких следов.

Рассмотрим пример копирования файлов — насколько быстрее это будет происходить в tmpfs по сравнению с дисками.

Создадим точку монтирования:

Создадим виртуальную файловую систему размером 20 Гигабайт в оперативной памяти:

sudo mount -t tmpfs -o size=20g tmpfs /tmp/mytmpfs

Скопируем туда файл размером в несколько Гигабайт:

cp /mnt/disk_d/Vuse/Space.Cop.2016.L2.BDRip.720p.mkv /tmp/mytmpfs

Проверим, сколько времени понадобится для создания копии этого файла в оперативной памяти:

time cp /tmp/mytmpfs/Space.Cop.2016.L2.BDRip.720p.mkv /tmp/mytmpfs/copy.mkv

real 0m1,403s user 0m0,020s sys 0m1,381s

Понадобилось совсем немного времени — примерно полторы секунды.

А теперь сделаем копию этого же файла на жёстком диске:

time cp /mnt/disk_d/Vuse/Space.Cop.2016.L2.BDRip.720p.mkv /mnt/disk_d/Vuse/copy.mkv

real 0m14,463s user 0m0,065s sys 0m4,041s

Понадобилось 14 секунд — в 10 раз больше времени.

Итак, используя tmpfs можно добиться максимальной скорости доступа к файлам.

Смотрите также

Связанные статьи:

Источник

How to Create a Ramdisk in Linux

A ramdisk is a volatile storage space defined in the RAM memory. Using this feature increases file processing performance ten times over the best SSD hard disks. Implementing a ramdisk is very advantageous for users whose tasks require significant amounts of hardware resources. In addition, media editors and gamers can enjoy this implementation.

A ramdisk is a volatile space, all information stored in it will be lost if the device is turned off or reboots.

In Linux, ramdisks can be created using the command mount and the filesystems tmpfs and ramfs. This tutorial shows how to create a ramdisk in Linux using both of them.

Tmpfs and Ramfs:

Tmpfs: Tmpfs is a temporary file system stored in the RAM memory (and/or swap memory). By specifying this file system with the argument -t of the command mount, you can assign limited memory resources to a temporary file system. As a result, applications stored in this filesystem will perform several times faster than they would on conventional storage devices, including cssd devices.

Ramfs: Ramfs is similar to Tmpfs, but the user can’t ensure a limit, and the allocated resource grows dynamically. If the user doesn’t control the ramfs consumption, ramfs will keep using all the memory until hanging or crashing the system.

Tmpfs vs. Ramfs: There is no notable difference between the performance of tmpfs and its predecessor ramfs. The reason behind ramfs being replaced by tmpfs is the unlimited RAM consumption risk by ramfs which may lead to a system crash.

Another advantage of tmpfs over ramfs is the ability to use the swap space while ramfs are limited to hardware memory.

How to Create a Ramdisk in Linux Using Tmpfs:

First, let’s see the free memory we can use before creating a tmpfs mount point. To check the available ram and swap, you can use the command free. To print the results in gigabytes, you can add the argument –giga, as shown in the example below:

As you can see in the output above, I have two physical GB and two on the swap space.

Now, create a mount point under the directory /mnt using the command mkdir as shown in the example below. The mount point name choice is arbitrary. If you are going to use the ramdisk for a specific application, you can name the mount point after it. In the example below I call it /mnt/tmp:

Now you can create the ramdisk using the mount command. The following example shows how to create a ramdisk using tmpfs on 2GB Ram/Swap, on the mount point /mnt/tmp.
The -t (type) argument allows to specify the file system (in this case, tmpfs). The -o (options) argument is used to define the space for the ramdisk.

The ramdisk was created under /mnt/tmp.

SSD vs. Tmpfs:

I copied an Ubuntu image from a user’s home directory to the root directory in the following screenshot.

Using the command time to display timing, you can see the copying process took 0:55.290s

In the following screenshot, you can see how copying the same Ubuntu iso image to the ramdisk takes 0:9.424s:

As you can see, the difference is titanic, and the ramdisk is very advantageous for tasks with large amounts of file writing.

To remove the ramdisk, just unmount it by running the following command and replacing tmp with your mount point:

Creating a Ramdisk in Linux Using Ramfs:

The procedure to create a ramdisk using ramfs is the same as with tmpfs. The following command will create a dynamic ramdisk on the mount point /mnt/tmp.

Tmpfs vs. Ramfs:

Now let’s test the ramfs performance against tmpfs, and let’s see what happens when each ramdisk type reaches the defined limit.

In the first example, I will create a 2GB ramdisk using tmpfs, and I will try to copy a bigger iso inside:

As you can see, the cp returned an error because the ramdisk space isn’t enough for the iso image. I only assigned 2GB for the ramdisk.

Now, see what happens when I do the same procedure using ramdisk:

As you can see, the ramfs kept writing into /mnt/tmp even though I have defined a 2GB limit. This is ramfs disadvantage because it may hang a system by consuming all its RAM memory. On the contrary, tmpfs is limited to the memory amount we define.

You can also see in the output that the copying task was done within 0:9.624s, almost the same performance shown by tmpfs in the test against SSD.

Note: The same iso image was used.

Conclusion

Creating a ramdisk is a one-minute process with significant benefits for any user who needs to process big files. The reading and writing speed increase exponentially over the best hard disks in the market. Portable software can be executed from a ramdisk, though changes won’t be persistent. This implementation is being highly appreciated by media editors whose tasks require long periods of media conversion.

Using ramfs may be risky if the system runs out of resources. That’s why tmpfs became the first method.

I hope this tutorial to create a ramdisk in Linux was useful. Keep following Linux Hint for more Linux tips and tutorials.

About the author

David Adams

David Adams is a System Admin and writer that is focused on open source technologies, security software, and computer systems.

Источник

How to place / store a file in memory on linux?

I have read somewhere that one can put a file on a linux system into memory, and loading it will be superfast. How do I do this? How do I verify the file is loaded from memory?

2 Answers 2

On Linux, you probably already have an tmpfs filesystem that you can write to at /dev/shm .

$ >/dev/shm/foo $ df /dev/shm/foo Filesystem 1K-blocks Used Available Use% Mounted on tmpfs 224088 0 224088 0% /dev/shm 

This may use swap, however. For a true ramdisk (that won’t swap), you need to use the ramfs filesystem.

mount ramfs -t ramfs /mountpoint 

Will this be then available to apache/php? I am interested in using this for a chat app and plan to save/retrieve recent lines from memory to avoid HDD write/read overhead.

It’s called a ramdisk. You can simply mount your RAM as follows:

mount tmpfs -t tmpfs -o size=2G 

This creates a ramdisk of 2 GiB. For more information see man mount and search for tmpfs .

You must log in to answer this question.

Related

Hot Network Questions

Subscribe to RSS

To subscribe to this RSS feed, copy and paste this URL into your RSS reader.

Site design / logo © 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA . rev 2023.7.14.43533

Linux is a registered trademark of Linus Torvalds. UNIX is a registered trademark of The Open Group.
This site is not affiliated with Linus Torvalds or The Open Group in any way.

By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.

Источник