How to make a temporary file in RAM?
I have a script that will pipe its output to |tee scriptnameYYMMDD.txt . After each cycle of the for loop in which the output is generated, I’ll be reversing the file contents with tac scriptnameYYYYMMDD.txt > /var/www/html/logs/scriptname.txt so that the log output is visible in a browser window with the newest lines at the top. I’ll have several scripts doing this in parallel. I’m trying to minimize the disk activity, so output from |tee scriptnameYYYYMMDD.txt to a RAMdisk would be best. mktemp creates a file in the /tmp folder, but that doesn’t appear to be off-disk.
It need to be a ramfs or a tmpfs (second is better). Check if a such fs is mounted on your system already, if yes you can use that. If no, you need to mount it.
you might consider incrementally tail ing the files (or initiating it via CGI upon request or something) rather than tac ing the whole thing.
This seems to work in testing (sorry for the diminished formatting): TEMPPATH=»/ramdisk» LOGPATH=»/var/www/html/log» . echo
Update on my progress on this: I have a 1 GB tmpfs volume mounted at /ramdisk. I have files coming into user home folders via FTP. They are moved to a subfolder /ramdisk/queues/xyz/ to process and are deleted afterwards. On startup, I have a script that re-creates my needed directory structure under /ramdisk. This should result in minimal disk activity for the incoming files. So far the only other way I see to reduce disk I/O for these scripts would be to create those user folders in a tmpfs created on startup, before any files come in via FTP to those user folders. Thanks all.
4 Answers 4
You can mount a tmpfs partititon and write the file there:
mount -t tmpfs -o size=500m tmpfs /mountpoint This partition now is limited to 500 MB. If your temporary file grows larger than 500 MB an error will occur: no space left on device . But, it doesn’t matter when you specify a larger amount of space than your systems RAM has. tmpfs uses swap space too, so you cannot force a system crash, as opposed to ramfs .
You can now write your file into /mountpoint :
command | tee /mountpoint/scriptnameYYYYMMDD.txt size=[num]% is also fine — a tmpfs can be limited to a percentage of available memory. It is usually easier that way if the commands are not always targeted for the same (real/virtual) machine. Also handy is that a tmpfs doesn’t drop contents w/ -o remount,size=newsize .
Since the question revolves around a bash script, which we have no reason to assume is run by root, can you specify whether one could get around mount not allowing non-root users to use -t or -o ? That is, how can one build scripts that use tmpfs (or RAM in any other fashion) to save on disk operations, if a suitable filesystem doesn’t already exist on the system?
The following answer was discovered by investigating the previous answers and the info in this question here and would not have been found without them. Cudos to them.
On my linuxmint system (and I would assume most ubuntu based systems and possibly debian based too) there is a user owned tmpfs mounted automatically on /run/user/1000/
11:41:11 jesse@Limbo:~$ df -T Filesystem Type 1K-blocks Used Available Use% Mounted on udev devtmpfs 15904812 4 15904808 1% /dev tmpfs tmpfs 3184120 1700 3182420 1% /run /dev/sdb2 ext4 14248880 11464788 2037240 85% / none tmpfs 4 0 4 0% /sys/fs/cgroup none tmpfs 5120 0 5120 0% /run/lock none tmpfs 15920584 848 15919736 1% /run/shm none tmpfs 102400 12 102388 1% /run/user /dev/sdb3 ext4 100861352 90755700 4959136 95% /mnt/data
Under /run/user/ there is a directory for each normal user on the system
12:07:35 jesse@Limbo:~$ ls -l /run/user total 0 drwx------ 2 root root 40 Aug 7 09:50 0 drwx------ 8 jesse jesse 180 Aug 7 11:38 1000 These directories are named after their respective user’s ids. We can get the user id with id -u see man id for details on this command.
12:07:43 jesse@Limbo:~$ ls -l /run/user/$(id -u) total 0 drwx------ 2 jesse jesse 60 Aug 7 09:50 dconf dr-x------ 2 jesse jesse 0 Aug 7 09:50 gvfs drwx------ 2 jesse jesse 80 Aug 7 09:50 pulse lrwxrwxrwx 1 root root 17 Aug 7 09:50 X11-display -> /tmp/.X11-unix/X0 We can then use the mktemp command with the —tmpdir option to create temp files and directories in this tempfilesystem thus creating tempfiles in RAM.
Following the advice given here I create a temp directory first and then create my temp files in that:
mydir=$(mktemp -dt "$(basename $0).XXXXXXXX" --tmpdir=/run/user/$(id -u)) to create a temp directory /run/user/1000/bash.w42BYxbG/ then
myfile=$(mktemp -t "$(basename $0).XXXXXXXX" --tmpdir=$mydir) to create a tempfile in it.
This makes cleaning up these file easy since all I have to do is rm -r $mydir .
By default all these files are owned and readable only by the user who created them.
Note: The $(basename $0) portion of the command extracts the name of the script/process that executed mktemp. If I have a script /home/jesse/scripts/myScript.sh then $(basename $0) returns myScript.sh when executed by this script. Thus the above commands would create /run/user/1000/myScript.sh.w42BYxbG/ and /run/user/1000/myScript.sh.w42BYxbG/myScript.sh.BCzSmq06 respectively.
How to place / store a file in memory on linux?
I have read somewhere that one can put a file on a linux system into memory, and loading it will be superfast. How do I do this? How do I verify the file is loaded from memory?
2 Answers 2
On Linux, you probably already have an tmpfs filesystem that you can write to at /dev/shm .
$ >/dev/shm/foo $ df /dev/shm/foo Filesystem 1K-blocks Used Available Use% Mounted on tmpfs 224088 0 224088 0% /dev/shm This may use swap, however. For a true ramdisk (that won’t swap), you need to use the ramfs filesystem.
mount ramfs -t ramfs /mountpoint Will this be then available to apache/php? I am interested in using this for a chat app and plan to save/retrieve recent lines from memory to avoid HDD write/read overhead.
It’s called a ramdisk. You can simply mount your RAM as follows:
mount tmpfs -t tmpfs -o size=2G This creates a ramdisk of 2 GiB. For more information see man mount and search for tmpfs .
You must log in to answer this question.
Related
Hot Network Questions
Subscribe to RSS
To subscribe to this RSS feed, copy and paste this URL into your RSS reader.
Site design / logo © 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA . rev 2023.7.14.43533
Linux is a registered trademark of Linus Torvalds. UNIX is a registered trademark of The Open Group.
This site is not affiliated with Linus Torvalds or The Open Group in any way.
By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.
Writing memory of the traced process.
I am playing around with ptrace in linux. I am trying to write the memory of the traced process using /proc/pid/mem interface. the function I ma using for accomplish this task is :
void write_proc(pid_t child, unsigned long int addr) < char mem_file_name[100]; char buf[10]="hope"; int mem_fd; memset( (void*)mem_file_name, 0, 100); memset( (void *)buf, 0, 10); sprintf(mem_file_name, "/proc/%d/mem", child); mem_fd = open(mem_file_name, O_RDONLY); lseek(mem_fd, addr , SEEK_SET); if (write(mem_fd, buf, 5)
But I always get the error : Writing: Bad file descriptor. Is it possible to write the traced process using this method?
2 Answers 2
You are opening the file in read-only mode ( O_RDONLY ). I'd suggest trying again with O_RDWR instead:
mem_fd = open(mem_file_name, O_RDWR); However, from man proc it's not clear this will work:
/proc/[pid]/mem This file can be used to access the pages of a process's memory through open(2), read(2), and lseek(2). I was curious too, so I put together this example using just ptrace() directly:
#include #include #include #include #include #define SHOW(call) ( < int _ret = (int)(call); printf("%s ->%d\n", #call, _ret); if (_ret < 0) < perror(NULL); >>) char changeme[] = "This is a test"; int main (void) < pid_t pid = fork(); int ret; int i; union < char cdata[8]; int64_t data; >u = < "Hijacked" >; switch (pid) < case 0: /* child */ sleep(1); printf("Message: %s\n", changeme); exit(0); case -1: perror("fork"); exit(1); break; default: /* parent */ SHOW(ptrace(PTRACE_ATTACH, pid, 0, 0)); SHOW(ptrace(PTRACE_POKEDATA, pid, changeme, u.data)); SHOW(ptrace(PTRACE_CONT, pid, 0, 0)); wait(NULL); break; >return 0; > I'm not sure it should work. perhaps mmap -ing that file coul be better. However, O_RDWR is required at open time (but might not work for /proc/*pid/mem . I don't know. )
You are right. The only problem was the I was opening the file on read only mode. Copy and paste is the source of most bugs. AHAHAH! Thanks!! However, PTRACE_POKEDATA is not a goo solution you can write only 4 (x32) or 8 (x64) per time. The best solution for this problem is : man7.org/linux/man-pages/man2/process_vm_readv.2.html
ptrace(2) is a very arcane syscall, only used by debuggers and the like.
For sure, the documented PTRACE_POKEDATA request to ptrace should work (when the traced process is stopped) and gives you the ability to write into the memory of the traced process. I don't know if writing (or mmap -ing) to /proc/$pid/mem should work or not.
Googling on linux write /proc /mem give me notably this which suggests that /proc/$pid/mem was designed to be read-only, but might have been made writable in recent kernels. But recent Documentation/filesystems/proc.txt from kernel source tree don't say much.
I would be cautious about writing on /proc/$pid/mem ; if it works (and it might not) it probably is very kernel version specific.
Perhaps mmap -ing some segments of that /proc/$pid/mem file does work (but I don't know). Have you tried that?
In contrast, PTRACE_POKEDATA should work (it has existed in SunOS and many other Unixes before Linux existed). Of course, it is fairly slow.
How to write file to RAM on Linux
A program that I can not modify writes it's output to a file provided as an argument. I want to have the output to go to RAM so I don't have to do unnecessary disk IO. I thought I can use tmpfs and "trick" the program to write to that, however not all Linux distros use tmpfs for /tmp, some mount tmpfs under /run (Ubuntu) others under /dev/shm (RedHat). I want my program to be as portable as possible and I don't want to create tmpfs file systems on the user's system if I can avoid it. Obviously I can do df | grep tmpfs and use whatever mount that returns, but I was hoping for something a bit more elegant. Is it possible to write to a pseudo terminal or maybe to /proc somewhere?
Since you've tagged this question python , how about stringIO ? It lets you use a string variable as a file, basically.
Is this for a program you're writing, or one that's already been written that you can't modify? Your question is contradictory on the matter.
I'm writting a python program that's calling another program repeatedly. StringIO won't work, the program I'm calling from my script is taking the location of the file where to write, not an actual file handle.
PyFilesystem might be a good fit then. docs.pyfilesystem.org/en/latest/tempfs.html#module-fs.tempfs Basically you are trying to create a userspace ramdisk. I would still try to lookup a mounted tmpfs though. (e.g. mount | egrep -wi "tmpfs" | egrep -iv "/sys" | egrep "755" )