Filename length limits on linux?
See the Wikipedia page about file systems comparison, especially in column Maximum filename length.
Here are some filename length limits in popular file systems:
BTRFS 255 bytes exFAT 255 UTF-16 characters ext2 255 bytes ext3 255 bytes ext3cow 255 bytes ext4 255 bytes FAT32 8.3 (255 UCS-2 code units with VFAT LFNs) NTFS 255 characters XFS 255 bytes @rahmanisback that’s right for filename limits, while path limits are usually defined by the OS, not FS (except for some strange FSes like iso or ntfs), and, on linux, are 4K
I’m shocked by these limits. Files per directory is unlimited, 4 billion files per volume, file sizes into the terabytes, volume sizes go to exabytes but we have a stupid limit of 255 bytes for file names?
It should also be mentioned that if you layer eCryptFS with filename encryption on top of these file systems (as installed in Ubuntu with encrypted home dir option) the effective maximum filename length will only be 143 characters. See: unix.stackexchange.com/a/32834/47938
I’ve read here that path length limit is in system headers. File name length limit is there too. On my system it’s file:
/usr/src/linux-headers-2.6.38-10/include/linux/limits.h #define NAME_MAX 255 /* # chars in a file name */ #define PATH_MAX 4096 /* # chars in a path name including nul */ Sorry, but I’am new here an can’t even comment, save vote. The previous answer (by sfp) should be upped, as it answers the question completely, while the others are partially off. Again, sorry for going besides the rules, but I can’t be quiet when the best answer is at the bottom.
@DavidBalažic: Although true, PATH_MAX under linux is just a guideline, most of the underlying file systems do not have a limitation. This makes it hard to reference paths that greater than that size. I usually use «chunks» of PATH_MAX as a size.
I refer to other answers, please upvote them.
Are there any filename or path length limits on Linux?
Yes, filename and pathname lengths are limited by :
To dynamically get these properties:
- Use functions pathconf and fpathconf as proposed by Michael Aaron Safyan
- Create a filename (or pathname) longer and longer as explained by dogbane
- Use the command getconf as proposed by tim that is also available on Linux:
$ getconf NAME_MAX /mnt/sda2/ 255 $ getconf PATH_MAX /mnt/sda3/ 4096 And for the sake of saving time (and anchoring it to memory):
ext2, ext3, ext4, zfs: no pathname limits; 255 bytes filename limit.
Most programs are limited with absolute paths to PATH_MAX = 4096 , though. That can be worked around if your program is able to use relative paths and you change your working directory first.
It’s because various POSIX APIs such as getcwd and realpath (which you can reimplement in userspace code by reading the metadata for . and then changing to .. and repeating until you hit the filesystem root) rely on PATH_MAX . (Source)
Yes, however, awfully lot of misc library code is prone to break if your code changes current directory at will. A well made code doesn’t break but most of the existing code is not well made.
Those are file system name lengths. «linux» itself has some too. For instance, from bits/stdio_lim.h:
So since the extX filesystems have a lower filename limit than what’s defined in the kernel, you wouldn’t ever hit that limit, unless it also encompases pathnames, right?
that’s what it looks like to me. There’s also PATH_MAX for the path, which is 4096, so it would be hit before the «unlimited» path size on the exts. I’m not sure how the OS resolves its own internal restrictions and those of the FS, never had my arms in that deep. interesting question though.
4096 characters is a helluva path name. I’m sure it could be raised with a recompile, but honestly, /why would you need pathnames that long?/
I’m not sure you would need it or not. I view it more as a protection against malicious or negligent programs (I could easily see a script that behaves poorly and begins creating the same dir recursively. Actually, I’ve made that script, but it was redirecting a web site, not creating dirs. ).
There is no way to determine the maximum length of paths on Linux in a portable way. On my system:
$ getconf PATH_MAX / 4096 $ getconf _POSIX_PATH_MAX / 4096 But I can easily create paths much longer than 4096 characters. Instead see PATH_MAX as a lower bound. You are guaranteed to be able to create paths this long, but you might also be able to create much longer ones.
A simple portable way to find the maximum length empirically is to write a program which creates longer and longer directory chains, and see where it fails. You won’t know exactly why it fails (though you would hope for a suggestive human-readable error message) but you’ll know how far you can safely go. Remember to check for both individual directory length, relative pathname length, and absolute pathname length.
Also, e.g. the Python os.pathconf() module will have some answers; if the Python port is any good, they should be reasonable.
You can’t because some filesystems don’t impose any limits. It would sooner fail with an out of memory error which any program would have a hard time recovering from.
This is the correct answer, except this is due to @BjörnLindqvist comment. PATH_MAX is just a guideline, and 99% of files will probably be within that limit.
Is there a reasonble way to increase the file name limitation of 255 bytes?
It seems that the file name length limitation is 255 «characters» on Windows (NTFS), but 255 «bytes» on Linux (ext4, BTRFS). I am not sure what text encoding those file systems use for file names, but if it is UTF-8, one Asian character, such as Japanese, could take 3 or more bytes. So, for English, 255 bytes means 255 characters, but for Japanese, 255 bytes could mean a lot less characters, and this limitation could be problematic in some cases. Other than practically impossible method for a general user like modifying Linux file system/kernel etc, is there any practical way to increase the limitation so that I could have guaranteed 255-character file name capacity for Asian characters on Linux?
2 Answers 2
In many cases, the 255-byte limit is baked into the on-disk format; see for example Ext4 which only provides 8 bits to encode the name length. Thus, even if you could work around the kernel APIs’ limits, you wouldn’t be able to store anything longer than 255 bytes anyway.
You would therefore have to come up with a name storage extension (for example, VFAT-style using multiple directory entries to store names which are too long, or 4DOS-style using a separate file to store the long names), and then you’re effectively creating a new file system.
TL/DR: there’s a way but unless you’re a kernel hacker/know C very well, there’s no way.
Detailed answer:
While glibc defines #define FILENAME_MAX 4096 on Linux which limits path length to 4096 bytes there’s a hard 255 bytes limit in Linux VFS which all filesystems must conform to. The said limit is defined in /usr/include/linux/limits.h :
/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ #ifndef _LINUX_LIMITS_H #define _LINUX_LIMITS_H #define NR_OPEN 1024 #define NGROUPS_MAX 65536 /* supplemental group IDs are available */ #define ARG_MAX 131072 /* # bytes of args + environ for exec() */ #define LINK_MAX 127 /* # links a file may have */ #define MAX_CANON 255 /* size of the canonical input queue */ #define MAX_INPUT 255 /* size of the type-ahead buffer */ #define NAME_MAX 255 /* # chars in a file name */ #define PATH_MAX 4096 /* # chars in a path name including nul */ #define PIPE_BUF 4096 /* # bytes in atomic write to a pipe */ #define XATTR_NAME_MAX 255 /* # chars in an extended attribute name */ #define XATTR_SIZE_MAX 65536 /* size of an extended attribute value (64k) */ #define XATTR_LIST_MAX 65536 /* size of extended attribute namelist (64k) */ #define RTSIG_MAX 32 #endif And here’s a piece of code from linux/fs/libfs.c which will throw an error in case you dare use a filename length longer than 255 chars:
/* * Lookup the data. This is trivial - if the dentry didn't already * exist, we know it is negative. Set d_op to delete negative dentries. */ struct dentry *simple_lookup(struct inode *dir, struct dentry *dentry, unsigned int flags) < if (dentry->d_name.len > NAME_MAX) return ERR_PTR(-ENAMETOOLONG); if (!dentry->d_sb->s_d_op) d_set_d_op(dentry, &simple_dentry_operations); d_add(dentry, NULL); return NULL; > So, not only you’ll have to redefine this limit, you’ll have to rewrite filesystems source code (and disk structure) to be able to use it. And then outside of your device, you won’t be able to mount such a filesystem unless you use its extensions to store very long filenames (like FAT32 does).
Copy files with long names [closed]
I am on linux ubuntu 16 and I need to copy around 400gb(some 100.000 files) of data from my HDD to my SSD. This I can’t do because around 1000 of those files have «too long name» and I can’t just skip them since it will take a long time to find them. Any program that copies files with long names?
What filesystem is it that you’re using that does not support these names? MSDOS? Would you be able to put the files in a tar archive and transfer that instead?
What filesystems are they? What are you using to copy the files? If something like cp on the command line, shouldn’t you get an error printed out for them?
It’s not terribly difficult to compare two lists of filenames. Try cd /path1 && find . -type f | sort >/tmp/list1 then repeat for path2 and list2 . Use sdiff /tmp/list1 /tmp/list2 to see differences or comm -3 /tmp/list1 /tmp/list2 to see files in list1 that aren’t in list2 and v.v.
If there’s a limit on file name length, it’s probably due to the filesystem(s), not to the tool you used. (I say “probably” because you didn’t say which tool you used, so it’s possible that you found some weird unusual tool that has a limitation.) What are examples of files whose name is too long, and what filesystem type are you copying them to?
Do you use home folder encryption? I’m asking because the home folder encryption Ubuntu uses prohibits files with a name longer than 144ish characters, whereas usually on ext<2, 3, 4>, you can use file names with up to 255 (not 255ish) characters.
1 Answer 1
Original (wrong) answer
Cool guys told, that rsync works like a charm:
rsync -auv --exclude '.svn' --exclude '*.pyc' source destination UPD: with script
ok, other cool guys told, that rsync is not a solution, when filesystem itself doesn’t support long names. I’ll take note, that rsync is not a metaphysical low-level super secret tool, made by gods (there are a lot of such tools for Windows, btw;)
So, here is a short python script (python 2.7 is installed by default on Ubuntu, as I know), which copies all files from SRC to DST , and will print names of files, caused errors (including, long names)
import os import sys import shutil def error_on_dir(exc, dest_dir): print('Error when trying to create DIR:', dest_dir) print(exc) print() def error_on_file(exc, src_path): print('Error when trying to copy FILE:', src_path) print(exc) print() def copydir(source, dest, indent = 0): """Copy a directory structure overwriting existing files""" for root, dirs, files in os.walk(source): if not os.path.isdir(root): os.makedirs(root) for each_file in files: rel_path = root.replace(source, '').lstrip(os.sep) dest_dir = os.path.join(dest, rel_path) dest_path = os.path.join(dest_dir, each_file) try: os.makedirs(dest_dir) except OSError as exc: if 'file exists' not in str(exc).lower(): error_on_dir(exc, dest_dir) src_path = os.path.join(root, each_file) try: shutil.copyfile(src_path, dest_path) except Exception as exc: # here you could take an appropriate action # rename, or delete. # Currently, script PRINTS information about such files error_on_file(exc, src_path) if __name__ == '__main__': arg = sys.argv if len(arg) != 3: print('USAGE: python copy.py SOURCE DESTINATION') copydir(arg[1], arg[2])