How to create an .IMG image of a disc (sd card) without including free space?
but if the disk is of 32GB with only 4GB used, the 32GB image file is waste of space-time. Is there any way or tool to create images with only valid data?
7 Answers 7
Pretty good and simple way to deal with this is simply pipe it via gzip, something like this:
# dd if=/dev/sdb | gzip > backup.img.gz This way your image will be compressed and most likely unused space will be squeezed to almost nothing.
You would use this to restore such image back:
# cat backup.img.gz | gunzip | dd of=/dev/sdb One note: if you had a lot of files which were recently deleted, image size may be still large (deleting file does not necessarily zeroes underlying sectors). You can wipe free space by creating and immediately deleting large file containing zeros:
# cd /media/flashdrive # dd if=/dev/zero of=bigfile bs=1M # let it run and quit by disk full error # rm bigfile Well, you can simply tar cvfz mybackup.tar.gz /media/flashdrive . It will capture everything except for bootable USB drives they will lose bootable status
Yes, tar would lose MBR, and you will need to use boot-repair to make drive bootable again. But, other than that, tar should preserve everything and OS should be fully usable (but be sure to run tar as root!).
You can usually zero out free space on modern sdcards using ‘fstrim’, this has the advantage of being many times faster, and also avoids flash wear.For fstrim support on VFAT partitions, you will need at least Linux kernel version 4.19. Unfortunately, many USB SD Card reader/writers don’t support discard/trim. If you wish to try this, try fstrim -v /path-to-mounted-filesystem/ . I use this during production image writing on an embedded air quality sensor which I work on professionally.
- Copy all the files from all the partitions preserving meta data mkdir -p myimage/partition1 mkdir myimage/partition2 sudo cp -rf —preserve=all /media/mount_point_partition1/* myimage/partition1/ sudo cp -rf —preserve=all /media/mount_point_partition2/* myimage/partition2/
- Extract the MBR sudo dd if=/dev/sdX of=myimage/mbr.img bs=446 count=1 replace /dev/sdX with the corresponding device.
- Partition the destination disk into partitions with sizes greater than copied data and should be of the same format and same flags using gparted . Google how to partition a disk.
- Mount the freshly formatted and partitioned disk. On most computers, you just need to connect the disk and you can find the mounted partitions in /media folder.
- Copy the previously copied data to destination partitions using following commands sudo cp -rf —preserve=all myimage/partition1/* /media/mount_point_partition1/ sudo cp -rf —preserve=all myimage/partition2/* /media/mount_point_partition2/
- Copy back the MBR sudo dd if=myimage/mbr.img of=/dev/sdX bs=446 count=1
Has anyone tried this and confirm that it works? Otherwise, I would supply only one of -r or -R to cp : they are the same switch.
I’ve tried this and it worked. Made a clone of my raspberry pi’s boot SD card. The card had 2 partitions, one fat and one ext4. I have dumped those with commands provided, then I dd’ed entire SD card with zeroes, created new partiion table, created my partitions which had different from previous parameters: size, spacimg, labels. And after that I restored it from dumps and it worked.
This works well, but you can also just resize partitions using gparted in one step. Backing up is still a good idea though.
When you say MBR you probably mean Bootloader that is 446 bytes in size — amount you’re extracting. The MBR actually is 512 bytes .
Using the bs and count parameters of dd , you can limit the size of the image, as seen in step 2 of answer 1665017.
You may already know what size image you want to create. If not, you can get a good idea from df :
Substitute / with a space-separated list of all the mount points relating to the disk partitions.
A more accurate way might be to use fdisk or your preferred partition editor and get busy with a calculator.
$ fdisk -l /dev/mmcblk0 Disk /dev/mmcblk0: 7.4 GiB, 7948206080 bytes, 15523840 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x00057540 Device Boot Start End Sectors Size Id Type /dev/mmcblk0p1 2048 186367 184320 90M c W95 FAT32 (LBA) /dev/mmcblk0p2 186368 3667967 3481600 1.7G 5 Extended /dev/mmcblk0p5 188416 3667967 3479552 1.7G 83 Linux Total used space in bytes = end sector of last partition X sector size (here that’s 3667967 x 512).
Total used space in GB = total used space in bytes / 1024 3 (here that’s 1.749023 GB).
If you decide, for example, that your image should be exactly 2 GB, the following command will do that:
dd if=/dev/mmcblk0 of=/path/to/pi_updated.img bs=1M count=2048 The resulting image will also include the random noise beyond the greatest extent of the last partition.
Don’t forget to sudo the above commands if your account doesn’t already have sufficient privileges.
For my purposes, I don’t need an image that is perfectly trimmed down to the last bit of data so when the real size is 1.75 GB then a 2 GB image is near enough for me. This cuts out the other 6 GB (or 30 GB or whatever the device has spare) of unused space that would otherwise be in the image.
I have seen advice in many places that dd should not be performed on a mounted partition and I followed that because it seems intuitively correct; it does seem rather like trying to sketch yourself making a sketch in a mirror with the sketch you’re making also visible in the sketch. It’s a bit sketchy.
IMHO, while calculating, the sector size should be based one End+1 instead of End. In your case, it should be 3667968.
After trying multiple different methods, I found the following article:
It’s created to shrink and resize (on first boot) a raspberry-pi image but could be easily adjusted for any other Linux distribution. I successfully got it working with Debian 9 on a custom arm based chip.
The rc.local created by the pishrink script first uses raspi-config to resize the rootfs and then falls back to a method using parted (which I had to install ahead of time on my machine). I commented out the section of code where raspi-config .
My sd-card image was shrunk from 15 GB to 1.1 GB. I flashed the shrunk sd-card image with etcher. Took less than 5 mins as opposed to over half an hour for the full 15 GB image
If you have a big SD card 16 GB, 32 GB etc but you want to save space with backup you can use:
sudo apt-get install gnome-disk-utility Open disk utility to check witch letter is your usb drive actually has:
In my case a 32GB SD card with Raspbian image on it recognised as: /dev/sde
sudo dd bs=4M status=progress if=/dev/sde | gzip > \ /you-selected-full-path-here/raspberry-images/`date +%Y%m%d`_rpi_image_backup.gz status=progress gives you progress bar indication | gzip > compresses the 32 GB total size and not writing the empty space from the 32 GB `date +%Y%m%d` writes today date in the filename And the size is only 3.5GB. If you want to write this image to a new SD card use:
Also you can write this image made from 32 GB to a 16 GB or 8 GB disk, it is not complaining that the image is too large anymore.
I tried your steps for my 64 GB SD card which has 2 partitions. In total their size is about 7GB. However, after several minutes I see the process is not finished and the size of the created image is more than 25GB. I did again with gzio -9 and the result is almost the same. Why don’t I have an 7GB image?
Sparse (Shrunken) Image Cloning Instructions:
Plug in bootable Ubuntu persistent live USB with image file in usbdata partition into Single-Board Computer such as UP² Board. Boot to USB (default persistent live option). Launch Terminal and type in the following command (single line):
sudo dd if=/media/ubuntu/usbdata/ubuntu.img of=/dev/mmcblk0 bs=1M status=progress Wait 15-20 minutes. When completed, shutdown and disconnect USB. Boot to internal eMMC drive
Creating the Image:
Create uncompressed sparse image file from internal eMMC and save to flash drive
sudo dd if=/dev/mmcblk0 bs=1M status=progress | cp --sparse=always /dev/stdin /media/ubuntu/usbdata/ubuntu.img Extract uncompressed image file to internal eMMC
sudo dd if=/media/ubuntu/usbdata/ubuntu.img of=/dev/mmcblk0 bs=1M status=progress Create compressed image file from internal eMMC and save to Desktop
sudo dd if=/dev/mmcblk0 bs=1M status=progress | gzip > /home/user/Desktop/ubuntu.img.gz Extract compressed image file to internal eMMC
sudo gzip -cd /home/user/Desktop/ubuntu.img.gz | dd of=/dev/mmcblk0 bs=1M status=progress To modify an existing Linux image or drive from within Linux, plug in drive or mount the image:
sudo mkdir /media/root sudo mount -t ext4 -o loop,offset=537919488 /media/user/usbdata/ubuntu.img /media/root To determine the offset for your image or drive, run:
fdisk -l /media/user/usbdata/ubuntu.img Then, multiply block-size of 512 bytes by the start-block of the Linux partition, 1050624. 512*1050624 = 537,919,488
Also mount other system folders:
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /media/root$i; done Then cd into root folder of img and chroot:
Then you can update files, passwd, etc. To unmount, run:
for i in /dev /dev/pts /proc /sys /run; do sudo umount /media/root$i; done sudo umount /media/root Creating Ubuntu Persistent Live USB Image Follow these steps to flash a USB drive with a persistent live installation of Ubuntu: https://www.howtogeek.com/howto/14912/create-a-persistent-bootable-ubuntu-usb-flash-drive/
My solution to make an image of a customized raspios:
DIR="./img" DEVICE=/dev/sdb # declare custom copy functions copy() < echo "[$1] copy $2 to $3. " if [ "$1" == 'ext' ]; then sudo rsync -aHAXX \ --no-inc-recursive \ --numeric-ids \ --filter='-x security.selinux' \ --delete \ --exclude=\ --info=progress2 \ $2 \ $3 else sudo rsync -rtD \ --modify-window=1 \ --no-inc-recursive \ --delete \ --exclude= \ --info=progress2 \ $2 \ $3 fi > # create working directories mkdir -p $DIR// # mount initial device partitions sudo mount $1 $DIR/mount/boot sudo mount $2 $DIR/mount/rootfs # preserve partitions UUIDs PTUUID=0x`sudo blkid -p $DEVICE | sed -e 's/.*PTUUID="\([0-9a-f]*\)".*/\1/'` BOOTUUID=`sudo blkid -p $1 | sed -e 's/.* UUID="\([0-9a-zA-Z\-]*\)".*/\1/' | tr -d '-'` ROOTUUID=`sudo blkid -p $2 | sed -e 's/.* UUID="\([0-9a-zA-Z\-]*\)".*/\1/'` # copy contents from physical device to local folder copy fat $DIR/mount/boot/ $DIR/image/boot/ copy ext $DIR/mount/rootfs/ $DIR/image/rootfs/ umount $1 umount $2 # show contents size BOOTSIZE=`sudo du -hs $DIR/image/boot` ROOTSIZE=`sudo du -hs $DIR/image/rootfs` echo "boot is $BOOTSIZE" echo "root is $ROOTSIZE" # allocate the final .img file # change size as needed SIZE="5" echo "allocating $G image. " rm $DIR/$IMAGE fallocate -l $G $DIR/$IMAGE # create partitions echo "formatting image. " sed -e 's/\s*\([\+0-9a-zA-Z]*\).*/\1/' /dev/null o # create MBR partition scheme n # new partition p # primary 1 # partition #1 8192 # start block 532479 # end block t # define type 0c # W95 FAT32 (LBA) n # new partition p # primary 2 # partition #2 532480 # start block # last end block t # define type 2 # for partition #2 83 # linux x # advanced menu i # change disk identifier $PTUUID # identifier from original fstab r # return to normal menu w # write changes EOF # locally mount the .img echo "mounting image. " LOOP=`sudo losetup -fP --show $DIR/$IMAGE` echo "mounted image under $LOOP" ls -la $LOOP* echo "creating partition boot with FAT32" sudo mkfs.fat \ -F 32 \ -S 512 \ -i $BOOTUUID \ -n boot \ $p1 echo "creating partition rootfs with ext4" sudo mkfs.ext4 \ -b 4096 \ -U $ROOTUUID \ -L rootfs \ $p2 echo "mounting image loops" sudo mount $p1 $DIR/mount/boot sudo mount $p2 $DIR/mount/rootfs echo "copying contents to image" copy fat $DIR/image/boot/ $DIR/mount/boot/ copy ext $DIR/image/rootfs/ $DIR/mount/rootfs/ echo "unmounting image" unmount "$LOOP+" > /dev/null echo "removing loop" sudo losetup -d $