How to place / store a file in memory on linux?
I have read somewhere that one can put a file on a linux system into memory, and loading it will be superfast. How do I do this? How do I verify the file is loaded from memory?
2 Answers 2
On Linux, you probably already have an tmpfs filesystem that you can write to at /dev/shm .
$ >/dev/shm/foo $ df /dev/shm/foo Filesystem 1K-blocks Used Available Use% Mounted on tmpfs 224088 0 224088 0% /dev/shm This may use swap, however. For a true ramdisk (that won’t swap), you need to use the ramfs filesystem.
mount ramfs -t ramfs /mountpoint Will this be then available to apache/php? I am interested in using this for a chat app and plan to save/retrieve recent lines from memory to avoid HDD write/read overhead.
It’s called a ramdisk. You can simply mount your RAM as follows:
mount tmpfs -t tmpfs -o size=2G This creates a ramdisk of 2 GiB. For more information see man mount and search for tmpfs .
You must log in to answer this question.
Related
Hot Network Questions
Subscribe to RSS
To subscribe to this RSS feed, copy and paste this URL into your RSS reader.
Site design / logo © 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA . rev 2023.7.12.43529
Linux is a registered trademark of Linus Torvalds. UNIX is a registered trademark of The Open Group.
This site is not affiliated with Linus Torvalds or The Open Group in any way.
By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.
Файловая система в оперативной памяти — как пользоваться tmpfs
Файловая система tmpfs может найти повседневное применение в вашей деятельности, поскольку она невероятно быстрая и может помочь снизить нагрузку на ваше постоянное хранилище (особенно актуально тем, у кого Linux установлен на флешку или карту памяти).
tmpfs — это виртуальная файловая система, располагающаяся в оперативной памяти.
Средство tmpfs позволяет создавать файловые системы, содержимое которых находится в виртуальной памяти. Поскольку файлы в таких файловых системах обычно находятся в ОЗУ, доступ к файлам осуществляется очень быстро.
Файловая система создаётся автоматически при монтировании файловой системы с типом tmpfs с помощью следующей команды:
sudo mount -t tmpfs -o size=10M tmpfs /mnt/mytmpfs
Файловая система tmpfs имеет следующие свойства:
- Файловая система может использовать пространство подкачки, когда этого требует физическая нагрузка на память.
- Файловая система потребляет столько физической памяти и пространства подкачки, сколько требуется для хранения текущего содержимого файловой системы.
- Во время операции повторного монтирования (mount -o remount) размер файловой системы может быть изменён (без потери существующего содержимого файловой системы).
Если файловая система tmpfs размонтирована, её содержимое теряется (удаляется).
Вы можете скопировать в tmpfs файлы для максимально быстрого доступа. Это могут быть файлы баз данных или веб-сервера.
Ещё одна цель использования — снизить износ постоянного хранилища. Это не особенно актуально для жёсткого диска или твердотельного диска — современные модели при любом типе домашнего использования переживут нас. Но это может быть актуально, если система установлена на карту памяти. Вы можете разместить в оперативную память приложение, которое постоянно использует хранилище (часто обращается к файлам или непрерывно сохраняет файлы), тем самым ускорится работа этого приложения, а также всей системы за счёт снижения нагрузки на карту памяти.
Ещё одна возможная причина использование — незаметность, при работе в tmpfs всё будет происходить в оперативной памяти, а на постоянных хранилищах не останется никаких следов.
Рассмотрим пример копирования файлов — насколько быстрее это будет происходить в tmpfs по сравнению с дисками.
Создадим точку монтирования:
Создадим виртуальную файловую систему размером 20 Гигабайт в оперативной памяти:
sudo mount -t tmpfs -o size=20g tmpfs /tmp/mytmpfs
Скопируем туда файл размером в несколько Гигабайт:
cp /mnt/disk_d/Vuse/Space.Cop.2016.L2.BDRip.720p.mkv /tmp/mytmpfs
Проверим, сколько времени понадобится для создания копии этого файла в оперативной памяти:
time cp /tmp/mytmpfs/Space.Cop.2016.L2.BDRip.720p.mkv /tmp/mytmpfs/copy.mkv
real 0m1,403s user 0m0,020s sys 0m1,381s
Понадобилось совсем немного времени — примерно полторы секунды.
А теперь сделаем копию этого же файла на жёстком диске:
time cp /mnt/disk_d/Vuse/Space.Cop.2016.L2.BDRip.720p.mkv /mnt/disk_d/Vuse/copy.mkv
real 0m14,463s user 0m0,065s sys 0m4,041s
Понадобилось 14 секунд — в 10 раз больше времени.
Итак, используя tmpfs можно добиться максимальной скорости доступа к файлам.
Смотрите также
Связанные статьи:
Tmpfs¶
Tmpfs is a file system which keeps all of its files in virtual memory.
Everything in tmpfs is temporary in the sense that no files will be created on your hard drive. If you unmount a tmpfs instance, everything stored therein is lost.
tmpfs puts everything into the kernel internal caches and grows and shrinks to accommodate the files it contains and is able to swap unneeded pages out to swap space, if swap was enabled for the tmpfs mount. tmpfs also supports THP.
tmpfs extends ramfs with a few userspace configurable options listed and explained further below, some of which can be reconfigured dynamically on the fly using a remount (‘mount -o remount . ‘) of the filesystem. A tmpfs filesystem can be resized but it cannot be resized to a size below its current usage. tmpfs also supports POSIX ACLs, and extended attributes for the trusted.* and security.* namespaces. ramfs does not use swap and you cannot modify any parameter for a ramfs filesystem. The size limit of a ramfs filesystem is how much memory you have available, and so care must be taken if used so to not run out of memory.
An alternative to tmpfs and ramfs is to use brd to create RAM disks (/dev/ram*), which allows you to simulate a block device disk in physical RAM. To write data you would just then need to create an regular filesystem on top this ramdisk. As with ramfs, brd ramdisks cannot swap. brd ramdisks are also configured in size at initialization and you cannot dynamically resize them. Contrary to brd ramdisks, tmpfs has its own filesystem, it does not rely on the block layer at all.
Since tmpfs lives completely in the page cache and optionally on swap, all tmpfs pages will be shown as «Shmem» in /proc/meminfo and «Shared» in free(1). Notice that these counters also include shared memory (shmem, see ipcs(1)). The most reliable way to get the count is using df(1) and du(1).
tmpfs has the following uses:
- There is always a kernel internal mount which you will not see at all. This is used for shared anonymous mappings and SYSV shared memory. This mount does not depend on CONFIG_TMPFS. If CONFIG_TMPFS is not set, the user visible part of tmpfs is not built. But the internal mechanisms are always present.
- glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for POSIX shared memory (shm_open, shm_unlink). Adding the following line to /etc/fstab should take care of this:
tmpfs /dev/shm tmpfs defaults 0 0
tmpfs has three mount options for sizing:
The limit of allocated bytes for this tmpfs instance. The default is half of your physical RAM without swap. If you oversize your tmpfs instances the machine will deadlock since the OOM handler will not be able to free that memory.
The same as size, but in blocks of PAGE_SIZE.
The maximum number of inodes for this instance. The default is half of the number of your physical RAM pages, or (on a machine with highmem) the number of lowmem RAM pages, whichever is the lower.
Disables swap. Remounts must respect the original settings. By default swap is enabled.
These parameters accept a suffix k, m or g for kilo, mega and giga and can be changed on remount. The size parameter also accepts a suffix % to limit this tmpfs instance to that percentage of your physical RAM: the default, when neither size nor nr_blocks is specified, is size=50%
If nr_blocks=0 (or size=0), blocks will not be limited in that instance; if nr_inodes=0, inodes will not be limited. It is generally unwise to mount with such options, since it allows any user with write access to use up all the memory on the machine; but enhances the scalability of that instance in a system with many CPUs making intensive use of it.
tmpfs also supports Transparent Huge Pages which requires a kernel configured with CONFIG_TRANSPARENT_HUGEPAGE and with huge supported for your system (has_transparent_hugepage(), which is architecture specific). The mount options for this are:
never: disables huge pages for the mount
always: enables huge pages for the mount
within_size: only allocate huge pages if the page will be fully within i_size, also respect fadvise()/madvise() hints.
advise: only allocate huge pages if requested with fadvise()/madvise()
There is a sysfs file which you can also use to control system wide THP configuration for all tmpfs mounts, the file is:
This sysfs file is placed on top of THP sysfs directory and so is registered by THP code. It is however only used to control all tmpfs mounts with one single knob. Since it controls all tmpfs mounts it should only be used either for emergency or testing purposes. The values you can set for shmem_enabled are:
deny: disables huge on shm_mnt and all mounts, for emergency use
force: enables huge on shm_mnt and all mounts, w/o needing option, for testing
tmpfs has a mount option to set the NUMA memory allocation policy for all files in that instance (if CONFIG_NUMA is enabled) — which can be adjusted on the fly via ‘mount -o remount . ‘
use the process allocation policy (see set_mempolicy(2))
prefers to allocate memory from the given Node
allocates memory only from nodes in NodeList
prefers to allocate from each node in turn
allocates from each node of NodeList in turn
prefers to allocate memory from the local node
NodeList format is a comma-separated list of decimal numbers and ranges, a range being two hyphen-separated decimal numbers, the smallest and largest node numbers in the range. For example, mpol=bind:0-3,5,7,9-15
A memory policy with a valid NodeList will be saved, as specified, for use at file creation time. When a task allocates a file in the file system, the mount option memory policy will be applied with a NodeList, if any, modified by the calling task’s cpuset constraints [See CPUSETS ] and any optional flags, listed below. If the resulting NodeLists is the empty set, the effective memory policy for the file will revert to «default» policy.
NUMA memory allocation policies have optional flags that can be used in conjunction with their modes. These optional flags can be specified when tmpfs is mounted by appending them to the mode before the NodeList. See NUMA Memory Policy for a list of all available memory allocation policy mode flags and their effect on memory policy.
=static is equivalent to MPOL_F_STATIC_NODES =relative is equivalent to MPOL_F_RELATIVE_NODES
For example, mpol=bind=static:NodeList, is the equivalent of an allocation policy of MPOL_BIND | MPOL_F_STATIC_NODES.
Note that trying to mount a tmpfs with an mpol option will fail if the running kernel does not support NUMA; and will fail if its nodelist specifies a node which is not online. If your system relies on that tmpfs being mounted, but from time to time runs a kernel built without NUMA capability (perhaps a safe recovery kernel), or with fewer nodes online, then it is advisable to omit the mpol option from automatic mount options. It can be added later, when the tmpfs is already mounted on MountPoint, by ‘mount -o remount,mpol=Policy:NodeList MountPoint’.
To specify the initial root directory you can use the following mount options:
The permissions as an octal number
These options do not have any effect on remount. You can change these parameters with chmod(1), chown(1) and chgrp(1) on a mounted filesystem.
tmpfs has a mount option to select whether it will wrap at 32- or 64-bit inode numbers:
On a 32-bit kernel, inode32 is implicit, and inode64 is refused at mount time. On a 64-bit kernel, CONFIG_TMPFS_INODE64 sets the default. inode64 avoids the possibility of multiple files with the same inode number on a single device; but risks glibc failing with EOVERFLOW once 33-bit inode numbers are reached — if a long-lived tmpfs is accessed by 32-bit applications so ancient that opening a file larger than 2GiB fails with EINVAL.
So ‘mount -t tmpfs -o size=10G,nr_inodes=10k,mode=700 tmpfs /mytmpfs’ will give you tmpfs instance on /mytmpfs which can allocate 10GB RAM/SWAP in 10240 inodes and it is only accessible by root.
KOSAKI Motohiro, 16 Mar 2010