Linux create file big size

Quickly create a large file on a Linux system

How can I quickly create a large file on a Linux (Red Hat Linux) system? dd will do the job, but reading from /dev/zero and writing to the drive can take a long time when you need a file several hundreds of GBs in size for testing. If you need to do that repeatedly, the time really adds up. I don’t care about the contents of the file, I just want it to be created quickly. How can this be done? Using a sparse file won’t work for this. I need the file to be allocated disk space.

Ext4 has much better file allocation performance, since whole blocks of up to 100MB can be allocated at once.

People seem to be grossly ignoring the «sparse file won’t work with this», with their truncate and dd seeks below.

You should have defined what you meant by «for testing». Testing the writing speed of your hard disk? Testing what df will report? Testing an app that does something particular. The answer depends on what you want to test. Anyway I’m a bit late — I see now that it’s been years since your question 🙂

Just in case you are looking for a way to simulate a full partition, like I was, look no further than /dev/full

16 Answers 16

dd from the other answers is a good solution, but it is slow for this purpose. In Linux (and other POSIX systems), we have fallocate , which uses the desired space without having to actually writing to it, works with most modern disk based file systems, very fast:

fallocate -l 10G gentoo_root.img 

Is it possible that dd is internally using that already? If I do ‘dd if=/dev/zero of=zerofile bs=1G count=1’ on a 3.0.0 kernel, the write finishes in 2 seconds, with a write data rate of over 500 megabytes per second. That’s clearly impossible on a 2.5″ laptop harddrive.

This ( fallocate ) will also not work on a Linux ZFS filesystem — github.com/zfsonlinux/zfs/issues/326

In Debian GNU/Linux fallocate is part of the util-linux package. This tool was written by Karel Zak from RedHat and source code can be found here: kernel.org/pub/linux/utils/util-linux

This is a common question — especially in today’s environment of virtual environments. Unfortunately, the answer is not as straight-forward as one might assume.

dd is the obvious first choice, but dd is essentially a copy and that forces you to write every block of data (thus, initializing the file contents). And that initialization is what takes up so much I/O time. (Want to make it take even longer? Use /dev/random instead of /dev/zero! Then you’ll use CPU as well as I/O time!) In the end though, dd is a poor choice (though essentially the default used by the VM «create» GUIs). E.g:

dd if=/dev/zero of=./gentoo_root.img bs=4k iflag=fullblock,count_bytes count=10G 

truncate is another choice — and is likely the fastest. But that is because it creates a «sparse file». Essentially, a sparse file is a section of disk that has a lot of the same data, and the underlying filesystem «cheats» by not really storing all of the data, but just «pretending» that it’s all there. Thus, when you use truncate to create a 20 GB drive for your VM, the filesystem doesn’t actually allocate 20 GB, but it cheats and says that there are 20 GB of zeros there, even though as little as one track on the disk may actually (really) be in use. E.g.:

 truncate -s 10G gentoo_root.img 

fallocate is the final — and best — choice for use with VM disk allocation, because it essentially «reserves» (or «allocates» all of the space you’re seeking, but it doesn’t bother to write anything. So, when you use fallocate to create a 20 GB virtual drive space, you really do get a 20 GB file (not a «sparse file», and you won’t have bothered to write anything to it — which means virtually anything could be in there — kind of like a brand new disk!) E.g.:

fallocate -l 10G gentoo_root.img 

Источник

Как создать большой файл в Linux

Иногда вам может понадобиться создать большой файл в Linux в целях тестирования. Существует несколько способов сделать это. В этой статье мы рассмотрим, как создавать большие файлы в Linux. Вы можете использовать эти шаги для создания 100mb файла в Linux, или создания 1gb файла, или создания большого файла со случайными данными.

Как создать большой файл в Linux

Ниже описаны шаги по созданию большого файла в Linux. Допустим, вы хотите создать файл размером 100 Мб.

1. Использование dd

dd — самый распространенный способ создания больших файлов в Linux. Вот команда для создания файла data.img размером 100 Мб.

dd if=/dev/zero of=./data.img bs=1024 count=102400

Вот команда для создания файла размером 1 Гб.

dd if=/dev/zero of=./data.img bs=1024 count=1024000

Но dd — это программа копирования, которая записывает блок данных и поэтому требует много времени на операции ввода-вывода по мере увеличения размера файла. Это один из самых медленных методов создания больших файлов, который не рекомендуется использовать. Тем не менее, мы включили его сюда, поскольку это наиболее распространенный метод, используемый по умолчанию большинством системных администраторов.

2. Использование команды Truncate

Для создания больших файлов можно также использовать команду truncate. Это один из самых быстрых способов создания больших файлов. Truncate работает путем создания разреженного файла, то есть участка диска, который содержит идентичные данные, например, ноль. Он не хранит реальные данные и не выделяет место, а обманывает ОС, делая вид, что все данные есть. Реальное пространство выделяется только тогда, когда вы начинаете записывать данные в файл. Вот команда для создания файла размером 1 Гб с помощью команды truncate.

3. Использование fallocate

fallocate — это лучший метод для создания больших файлов. Он резервирует или выделяет указанное дисковое пространство для вашего файла, фактически ничего не записывая в него. Таким образом, в этом случае вы получаете файл нужного объема, а не разреженный файл, как в случае с truncate. Вот команда для создания файла размером 1 Гб с помощью fallocate.

Вот и все. В этой статье мы узнали, как создавать большие файлы в Linux.

Источник

How to Create a Large 1GB or 10GB File in Linux

Before we dive into the procedural steps needed to create large files in a Linux operating system environment, we must answer the WHY before we dwell on the HOW.

WHY create large files on a Linux operating system environment? Afterward, HOW do you create large files on a Linux operating system environment?

Why Create Large File on Linux

In Linux, you will find the need for dummy or sample files very useful in various operating system scenarios. You might need these files for testing or debugging OS-related functionalities or even projects.

For you to perfectly test or debug anything under Linux, you will need your dummy or sample files to exist in various sizes. Therefore, instead of wasting time populating a file with data to achieve the desired file size, you can achieve this objective with the mastery of a few Linux commands.

How to Create a Large File on Linux

Before we dwell on the HOW, consider this scenario: You have created or you have come across a promising Linux archive utility project/application. To test its performance and efficiency, you need instant access to a 1GB or 10GB file. It is under such circumstances that we reference system text manipulating commands to create our files.

We are going to primarily reference the dd command to demonstrate the creation of our large files. Additionally, you might need the assistance of the following listed commands.

• dd: For file image conversion and copying through cloning/creating/overwriting.
• df: For displaying your Linux system’s free disk space.
• du: For displaying your Linux system’s disk usage statistics.
• ls: For listing existing or created file sizes.

Using dd Command to Create 1GB or 10GB Files

The syntax for this command is highlighted below:

$ dd if=/dev/zero of=/path/to/file/storage/output.img [options]

Alternatively, the following detailed syntax reference also works.

$ dd if=/dev/zero of=YOUR_IMAGE_FILE_NAME bs=1 count=0 seek=File_Size_HERE

To make this article guide interesting, we are going to create a 10GB file on our Linux system. The command we are going to use is as simple as demonstrated below. Make sure your Linux system can accommodate the file size you want to create through the following df command.

On my end, I have 14GB of free disk space. It will be sufficient to accommodate the 10GB file we want to create.

To create the 10GB file on my Linux machine, I will run the following command.

$ dd if=/dev/zero of=10g.img bs=1 count=0 seek=10G

The next step is to verify that our created image file actually exists.

As highlighted on the above screen capture, the 10g.img image file does exist. The next step is to verify the size of this image file.

The size of our created file is indeed 10GB as per the above screen capture.

You can also right-click on this image file and accessing its properties will also highlight the expected size property.

The below du command implies that despite the existence of the created 10GB file size, the Linux operating system recognizes the disk usage of this 10GB file as zero since it’s a dummy binary image file (non-text file) and is yet to hold any sizable data.

You might also like to read the following related articles:

For whatever reason, you might need to create a dummy or test file under your Linux operating system environment, always consider the speed and efficiency of Linux’s inbuilt dd command to get things done. If you need to explore more options relating to the usage of this command, run man dd on your Linux terminal.

Источник

How to create a file with ANY given size in Linux?

I have read this question: How to create a file with a given size in Linux? But I havent got answer to my question. I want to create a file of 372.07 MB, I tried the following commands in Ubuntu 10.08:

dd if=/dev/zero of=output.dat bs=390143672 count=1 dd: memory exhausted 

390143672=372.07*1024*1024 Is there any other methods? Thanks a lot! Edit: How to view a file’s size on Linux command line with decimal. I mean, the command line ls -hl just says: ‘373M’ but the file is actually «372.07M».

4 Answers 4

Sparse file

dd of=output.dat bs=1 seek=390143672 count=0 

This has the added benefit of creating the file sparse if the underlying filesystem supports that. This means, no space is wasted if some of the pages (_blocks) ever get written to and the file creation is extremely quick.

Non-sparse (opaque) file:

Edit since people have, rightly pointed out that sparse files have characteristics that could be disadvantageous in some scenarios, here is the sweet point:

You could use fallocate (in Debian present due to util-linux ) instead:

fallocate -l 390143672 output.dat 

This still has the benefit of not needing to actually write the blocks, so it is pretty much as quick as creating the sparse file, but it is not sparse. Best Of Both Worlds.

Источник