How to make a temporary file in RAM?
I have a script that will pipe its output to |tee scriptnameYYMMDD.txt . After each cycle of the for loop in which the output is generated, I’ll be reversing the file contents with tac scriptnameYYYYMMDD.txt > /var/www/html/logs/scriptname.txt so that the log output is visible in a browser window with the newest lines at the top. I’ll have several scripts doing this in parallel. I’m trying to minimize the disk activity, so output from |tee scriptnameYYYYMMDD.txt to a RAMdisk would be best. mktemp creates a file in the /tmp folder, but that doesn’t appear to be off-disk.
It need to be a ramfs or a tmpfs (second is better). Check if a such fs is mounted on your system already, if yes you can use that. If no, you need to mount it.
you might consider incrementally tail ing the files (or initiating it via CGI upon request or something) rather than tac ing the whole thing.
This seems to work in testing (sorry for the diminished formatting): TEMPPATH=»/ramdisk» LOGPATH=»/var/www/html/log» . echo
Update on my progress on this: I have a 1 GB tmpfs volume mounted at /ramdisk. I have files coming into user home folders via FTP. They are moved to a subfolder /ramdisk/queues/xyz/ to process and are deleted afterwards. On startup, I have a script that re-creates my needed directory structure under /ramdisk. This should result in minimal disk activity for the incoming files. So far the only other way I see to reduce disk I/O for these scripts would be to create those user folders in a tmpfs created on startup, before any files come in via FTP to those user folders. Thanks all.
4 Answers 4
You can mount a tmpfs partititon and write the file there:
mount -t tmpfs -o size=500m tmpfs /mountpoint This partition now is limited to 500 MB. If your temporary file grows larger than 500 MB an error will occur: no space left on device . But, it doesn’t matter when you specify a larger amount of space than your systems RAM has. tmpfs uses swap space too, so you cannot force a system crash, as opposed to ramfs .
You can now write your file into /mountpoint :
command | tee /mountpoint/scriptnameYYYYMMDD.txt size=[num]% is also fine — a tmpfs can be limited to a percentage of available memory. It is usually easier that way if the commands are not always targeted for the same (real/virtual) machine. Also handy is that a tmpfs doesn’t drop contents w/ -o remount,size=newsize .
Since the question revolves around a bash script, which we have no reason to assume is run by root, can you specify whether one could get around mount not allowing non-root users to use -t or -o ? That is, how can one build scripts that use tmpfs (or RAM in any other fashion) to save on disk operations, if a suitable filesystem doesn’t already exist on the system?
The following answer was discovered by investigating the previous answers and the info in this question here and would not have been found without them. Cudos to them.
On my linuxmint system (and I would assume most ubuntu based systems and possibly debian based too) there is a user owned tmpfs mounted automatically on /run/user/1000/
11:41:11 jesse@Limbo:~$ df -T Filesystem Type 1K-blocks Used Available Use% Mounted on udev devtmpfs 15904812 4 15904808 1% /dev tmpfs tmpfs 3184120 1700 3182420 1% /run /dev/sdb2 ext4 14248880 11464788 2037240 85% / none tmpfs 4 0 4 0% /sys/fs/cgroup none tmpfs 5120 0 5120 0% /run/lock none tmpfs 15920584 848 15919736 1% /run/shm none tmpfs 102400 12 102388 1% /run/user /dev/sdb3 ext4 100861352 90755700 4959136 95% /mnt/data
Under /run/user/ there is a directory for each normal user on the system
12:07:35 jesse@Limbo:~$ ls -l /run/user total 0 drwx------ 2 root root 40 Aug 7 09:50 0 drwx------ 8 jesse jesse 180 Aug 7 11:38 1000 These directories are named after their respective user’s ids. We can get the user id with id -u see man id for details on this command.
12:07:43 jesse@Limbo:~$ ls -l /run/user/$(id -u) total 0 drwx------ 2 jesse jesse 60 Aug 7 09:50 dconf dr-x------ 2 jesse jesse 0 Aug 7 09:50 gvfs drwx------ 2 jesse jesse 80 Aug 7 09:50 pulse lrwxrwxrwx 1 root root 17 Aug 7 09:50 X11-display -> /tmp/.X11-unix/X0 We can then use the mktemp command with the —tmpdir option to create temp files and directories in this tempfilesystem thus creating tempfiles in RAM.
Following the advice given here I create a temp directory first and then create my temp files in that:
mydir=$(mktemp -dt "$(basename $0).XXXXXXXX" --tmpdir=/run/user/$(id -u)) to create a temp directory /run/user/1000/bash.w42BYxbG/ then
myfile=$(mktemp -t "$(basename $0).XXXXXXXX" --tmpdir=$mydir) to create a tempfile in it.
This makes cleaning up these file easy since all I have to do is rm -r $mydir .
By default all these files are owned and readable only by the user who created them.
Note: The $(basename $0) portion of the command extracts the name of the script/process that executed mktemp. If I have a script /home/jesse/scripts/myScript.sh then $(basename $0) returns myScript.sh when executed by this script. Thus the above commands would create /run/user/1000/myScript.sh.w42BYxbG/ and /run/user/1000/myScript.sh.w42BYxbG/myScript.sh.BCzSmq06 respectively.
раздел /tmp в оперативной памяти
В 21 веке оперативная память стоит копейки, и зачастую у читателя стоит не меньше 4ГБ RAM.
Можно с большой вероятностью сказать, что она мало когда полностью используется, и начинает заполняться swap раздел. Почему бы не использовать оперативку еще больше?
Воспользуемся таким способом, как монтирование директорий в tmpfs – файловую систему, находящуюся в оперативной памяти. Поместив директорию в область виртуальной памяти мы заметно ускорим доступ к файлам, которые будут туда записаны. Стоит учесть, что все файлы, помещенные в такую директорию будут утеряны после выключения питания или перезагрузки.
Для начала, давайте определим размер имеющейся в наличии оперативки, а вдруг мы не знаем. Заодно узнаем производителя и частоту, на которой она работает.
Воспользуемся утилитой, для просмотра информации из таблицы DMI(SMBIOS).
У меня получилось что-то, вроде:
# dmidecode 2.9 SMBIOS 2.6 present. Handle 0x0024, DMI type 17, 28 bytes Memory Device Array Handle: 0x0023 Error Information Handle: 0x0025 Total Width: 64 bits Data Width: 8 bits Size: 2048 MB Form Factor: DIMM Set: None Locator: DIMM0 Bank Locator: BANK0 Type: out of spec Type Detail: Synchronous Speed: 1066 MHz (0.9 ns) Manufacturer: Samsung Serial Number: 641DE5AC Asset Tag: Unknown Part Number: M471B5773CHS-CH9 Handle 0x0028, DMI type 17, 28 bytes Memory Device Array Handle: 0x0023 Error Information Handle: 0x0029 Total Width: 64 bits Data Width: 8 bits Size: 4096 MB Form Factor: DIMM Set: None Locator: DIMM1 Bank Locator: BANK0 Type: out of spec Type Detail: Synchronous Speed: 1066 MHz (0.9 ns) Manufacturer: Unknown Serial Number: 00000000 Asset Tag: Unknown Part Number: GR1333S364L9/4G Сложив колонки Size мы получим итоговый размер памяти.
Вы наверняка знаете, что в Linux используется устройство /dev/shm, которое представляет собой оперативную память. Обычно размер /dev/shm в два раза меньше размера RAM памяти
df -h /dev/shm Filesystem Size Used Avail Use% Mounted on tmpfs 3.0G 212K 3.0G 1% /dev/shm Но его можно изменить. Для временного изменения размера, воспользуйтесь командой mount:
mount -o remount,size=4G /dev/shm Для постоянного, воспользуйтесь изменением /etc/fstab. Нужны рутовые привилегии.
echo 'tmpfs /dev/shm tmpfs defaults,size=2g 0 0' >> /etc/fstab С размером доступной для оперирования оперативной памяти мы определились, осталось использовать её по назначению. Я монтирую RAM на /tmp и \$HOME/.cache Вот образец моего /etc/fstab:
tmpfs /tmp tmpfs defaults 0 0 tmpfs /home/insider/.cache tmpfs size=256M 0 0 После изменения fstab перезагружаемся или выполняем
и радуемся жизни, но всё же лучше перезагрузиться, т.к. монтирование потрет файлы в /tmp, что может привести к некорректной работе системы.
Работа системы должна немного ускориться.
How can I use RAM storage for the /tmp directory and how to set a maximum amount of RAM usage for it?
After seeing the comment by Anonymous on the question How is the /tmp directory cleaned up?, I found that it would be a great idea to implement on my system, since I have 16GB of RAM and I never used all of it.
My temporary files never get written to the disk. They get written to a RAM disk. I did put tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0 in /etc/fstab.
My question is: Can I set a maximum value for RAM Usage for /tmp ? And in that case, what would happen if the maximum amount got exceeded, would it write into the hard-disk drive? I have read a solution which states:
mkdir -p /tmp/ram sudo mount -t tmpfs -o size=512M tmpfs /tmp/ram/ But in my understanding, this won’t be a permanent solution. If I need it to be permanent, it has to be added to the /etc/fstab configuration file. If this is the correct solution, how can I transform that mount command into a line in /etc/fstab ?
2 Answers 2
You are absolutely right. The according fstab entry would look like this:
tmpfs /tmp tmpfs defaults,noatime,nosuid,nodev,noexec,mode=1777,size=512M 0 0 Please note:
As tmpfs gets filled up, it will behave as any physical hard drive by giving an «not enough space» error. While rebooting (and thus emptying the cache) will fix this, you may run into trouble when a single operation consumes more space to begin with than there’s space on tmpfs . In this case your computer will start to swap from ram to disk, which will make your system crawl to a halt, given you’ve got a swap partition to begin with, of course.
Considering this, a size of 512MB might be far too less nowadays, since much more ram is in existence in modern machines and it has become much cheaper. Since you’ve already got 16GB of ram, using the default value of half your ram for tmpfs should more than suffice for almost all scenarios. To use the default value, simply leave out the size=512M entry in your /etc/fstab file.
Another note:
You can quite as easily mount other system folders into ramdisk as well, such as
/var/log/apt (use only defaults,noatime without mode= or nosuid )
But beware: the same rules apply as above, running out of space might cause major trouble. E.g. imagine running out of space for /var/log/apt will render you unable to install any programs! Furthermore, loading /var/log folders into ramdisk will delete all your log files upon reboot, so you won’t be able to debug your system if anything unexpected happens. So use these settings at your own risk!
Editorial note: I removed the /run in tmpfs mount option since this folder and its subfolders are already mounted in tmpfs by default.