inodes, consumed space comparison for many small files (xfs, btrfs,ext4)
I have an ext4 partition (LVM on a VM) with a large amount of small files, which I have to extend every 3-4 months. Regarding the amount of space used by the inodes. Does one of the xfs, btrfs or ext4 file systems use less space? In other words, will switching to btrfs or xfs make a partition fill up with inodes slower then with ext4?
if you use ext4 and expect to be storing mostly small types, you should create it with mkfs.ext4 -t news for best results. Also, I’d suggest testing — create (on lvm, or loopback device for example) each of the filesytems in turn, and start copying your real files to it until it fills up. When it fills, do df -i (or find | wc -l ) to find which one managed to store most of your files — that way you’ll know for sure.
3 Answers 3
To start with your first question: yes, one of these filesystems uses less space. Even without further details it would be unlikely that they all use the exact same amount of space, given that they have different implementations. So one is bound to use less space than all of the others.
Btrfs has dynamic inode allocation, so there is no filling up like you have with the inode tables for ext4 (the size for which is set at ext4 filesystem creation time).
XFS is dynamic in a similar way, but has a limit (percentage of the filesystem that can be used for inodes), so there, whether you fill-up your inode allowance depends on the percentage set as well as the file count/file sizes
Thanks for the quick reply! Looks like XFS is the safest choice for now. Can this inode usage percentage be changed dynamically? and if not, is there a way to predict how much will be needed?
Yes, and be aware that everything depends on your needs:
Btrfs (pronounced as Butter FS, Better FS, or B-Tree FS)
Considering that the btrfs will be able for spanning over the multiple hard drives, it is a very good poit that it can support 16 times more drive space than the ext4. A maximum partition size of the btrfs file system is 16 exbibytes, as well as maximum file size is 16 exbibytes too.
Maximum Number of Files: 2**64
XFS
The XFS is a high-performance 64-bit journaling file system. XFS supports maximum file system size of 8 exbibytes for the 64-bit file system. Now, the RHEL 7.0 uses XFS as the default filesystem, including support for using XFS for the /boot partition.
Maximum Number of Files: 2**64
EXT4
The ext4 is well known because of bringing the speed improvements over ext3. The ext4 has some limits. The maximum file size is 16 tebibytes (which is roughly 17.6 terabytes). The largest volume/partition you can have with ext4 is 1 exbibyte. Like in the most modern file systems, it is a journaling file system that means that it will keep a journal of where the files are mainly located on the disk and of any other changes that happen to the disk. Regardless all of its features, it doesn’t support the transparent compression, the transparent encryption, or the data deduplication. The snapshots are supported technically, but such feature is experimental at best.
Maximum Number of Files: 4 billion
XFS vs Btrfs
XFS doesn’t have any RAID, while Btrfs RAID is not yet completely stable and is in its early days. XFS is more and more mature than Btrfs, but we cannot deny that Btrfs is powerful and a good-growing FileSystem.
For now, XFS is my choice — specially since it’s the default FS on RHEL 7 -, unless I really need Btrfs.
This has some good background information on the filesystems in general, but I don’t see the specific issue of small files being mentioned here.
@ilkkachu «large amount of small files» what does it mean? It’s all about inodes since files are all created via inodes and the inode is a data structure used to represent a filesystem object. So I think I have explained all the author needs, plus I’ve mentioned about maximum number of files.
did not address the question; on the topic, you just posted the theoretical maximum number of files, which can be tuned, while the question was more about how to tune that and which fs would be more optimized for the case stated
I suppose the problem you have is not the partition filling up with inodes per se, but running out of the number of inodes in the filesystem. ext4 reserves the inodes statically when the filesystem is created, but you can set the number with options to mkfs.ext4 :
-i bytes-per-inode
Specify the bytes/inode ratio. mke2fs creates an inode for every bytes-per-inode bytes of space on the disk. The larger the bytes-per-inode ratio, the fewer inodes will be created.-N number-of-inodes
Overrides the default calculation of the number of inodes that should be reserved for the filesystem (which is based on the number of blocks and the bytes-per-inode ratio). This allows the user to specify the number of desired inodes directly.
The manual states explicitly that the bytes per inode ratio cannot be changed after the FS is created, but the total number will scale to meet the ratio if the FS is resized.
You can also set the size of each inode. The default is 256 bytes on «most» filesystems, but can be reduced to 128 (the default for «small» filesystems). The extra space is used for storing extended attributes (e.g. SELinux labels), so if you don’t need those, it should be safe to reduce the size to the minimum.
-I inode-size
Specify the size of each inode in bytes. The inode-size value must be a power of 2 larger or equal to 128.
df -i should show the number of inodes allocated and used. With default options, one 30 GB partition I looked at had one inode for every 16 kB, but if your files are very small, you could set, say, -i 4096 to have one inode for every data block on the system.
If your files are smaller than 4096, you may want to reduce the file system block size too, since all regular files will require one full data block anyway. (That is, on ext4. I don’t know if other current filesystems do packing of small files.)
-b block-size
Specify the size of blocks in bytes. Valid block-size values are 1024, 2048 and 4096 bytes per block. If omitted, block-size is heuristically determined by the filesystem size and the expected usage of the filesystem (see the -T option).
mkfs.ext4 also has the -T option that can be used as a shorthand for some or all of these. The settings are in /etc/mke2fs.conf , which on my Debian makes e.g. mkfs.ext4 -T small equivalent to
mkfs.ext4 -b 1024 -I 128 -i 4096 Which might not be a bad set of options for lots of small files (and no xattrs.)
If your files are even smaller than one kB, a file system may not be the best way to save the data, but something like a database or an application specific system should perhaps be considered.
Millions of (small) text files in a folder
We would like to store millions of text files in a Linux filesystem, with the purpose of being able to zip up and serve an arbitrary collection as a service. We’ve tried other solutions, like a key/value database, but our requirements for concurrency and parallelism make using the native filesystem the best choice. The most straightforward way is to store all files in a folder:
$ ls text_files/ 1.txt 2.txt 3.txt - Write text file from web scrape (shouldn’t be affected by number of files in folder).
- Zip selected files, given by list of filenames.
My question is, will storing up to ten million files in a folder affect the performance of the above operations, or general system performance, any differently than making a tree of subfolders for the files to live in?
Related: How to fix intermittant “No space left on device” errors during mv when device has plenty of space. Using dir_index , which is often enabled by default, will speed up lookups but may limit the number of files per directory.
Why not try it quickly on a virtual machine and see what it’s like? With bash it’s trivial to populate a folder with a million text files with random characters inside. I feel like you’ll get really useful information that way, in addition to what you’ll learn here.
@JoshuaD: If you populate it all at once on a fresh FS, you’re likely to have all the inodes contiguous on disk, so ls -l or anything else that stat s every inode in the directory (e.g. bash globbing / tab completion) will be artificially faster than after some wear and tear (delete some files, write some new ones). ext4 might do better with this than XFS, because XFS dynamically allocates space for inodes vs. data, so you can end up with inodes more scattered, I think. (But that’s a pure guess based on very little detailed knowledge; I’ve barely used ext4). Go with abc/def/ subdirs.
Yea, I don’t think the test I suggested will be able to tell the OP «this will work», but it could definitely quickly tell him «this will not work», which is useful.
but our requirements for concurrency and parallelism make using the native filesystem the best choice What did you try? Offhand, I’d think even a lower-end RDBMS such as MySQL and a Java servlet creating the zip files on the fly with ZipOutputStream would beat just about any free Linux native filesystem — I doubt you want to pay for IBM’s GPFS. The loop to process a JDBC result set and make that zip stream is probably merely 6-8 lines of Java code.
5 Answers 5
This is perilously close to an opinion-based question/answer but I’ll try to provide some facts with my opinions.
- If you have a very large number of files in a folder, any shell-based operation that tries to enumerate them (e.g. mv * /somewhere/else ) may fail to expand the wildcard successfully, or the result may be too large to use.
- ls will take longer to enumerate a very large number of files than a small number of files.
- The filesystem will be able to handle millions of files in a single directory, but people will probably struggle.
One recommendation is to split the filename into two, three or four character chunks and use those as subdirectories. For example, somefilename.txt might be stored as som/efi/somefilename.txt . If you are using numeric names then split from right to left instead of left to right so that there is a more even distribution. For example 12345.txt might be stored as 345/12/12345.txt .
You can use the equivalent of zip -j zipfile.zip path1/file1 path2/file2 . to avoid including the intermediate subdirectory paths in the ZIP file.
If you are serving up these files from a webserver (I’m not entirely sure whether that’s relevant) it is trivial to hide this structure in favour of a virtual directory with rewrite rules in Apache2. I would assume the same is true for Nginx.
The * expansion will succeed unless you run out of memory, but unless you raise the stacksize limit (on Linux) or use a shell where mv is builtin or can be builtin (ksh93, zsh), the execve() system call may fail with a E2BIG error.
@StéphaneChazelas yes ok, my choice of words might have been better, but the net effect for the user is much the same. I’ll see if I can alter the words slightly without getting bogged down in complexity.
Just curious how you would uncompress that zip file if you avoid including the intermediate subdirectory paths in it, without running into the issues you discuss?
I’d recommend using zip -j — . and piping the output stream directly to the client’s network connection over zip -j zipfile.zip . . Writing an actual zipfile to disk means the data path is read from disk->compress->write to disk->read from disk->send to client. That can up to triple your disk IO requirements over read from disk->compress->send to client.
The ls command, or even TAB-completion or wildcard expansion by the shell, will normally present their results in alphanumeric order. This requires reading the entire directory listing and sorting it. With ten million files in a single directory, this sorting operation will take a non-negligible amount of time.
If you can resist the urge of TAB-completion and e.g. write the names of files to be zipped in full, there should be no problems.
Another problem with wildcards might be wildcard expansion possibly producing more filenames than will fit on a maximum-length command line. The typical maximum command line length will be more than adequate for most situations, but when we’re talking about millions of files in a single directory, this is no longer a safe assumption. When a maximum command line length is exceeded in wildcard expansion, most shells will simply fail the entire command line without executing it.
This can be solved by doing your wildcard operations using the find command:
or a similar syntax whenever possible. The find . -exec . \+ will automatically take into account the maximum command line length, and will execute the command as many times as required while fitting the maximal amount of filenames to each command line.