Cannot remove file: «Structure needs cleaning»
I have an external hard drive which is encrypted via LUKS. It contains an ext4 fs. I just got an error from rsync for a file which is located on this drive:
rsync: readlink_stat("/home/some/dir/items.json") failed: Structure needs cleaning (117) rm /home/some/dir/items.json rm: cannot remove ‘//home/some/dir/items.json’: Structure needs cleaning Does anyone know what I can do to remove the file and fix related issues with the drive/fs (if there are any)?
8 Answers 8
That is strongly indicative of file-system corruption. You should unmount, make a sector-level backup of your disk, and then run e2fsck to see what is up. If there is major corruption, you may later be happy that you did a sector-level backup before letting e2fsck tamper with the data.
I strongly suggest people to use GUIs like e.g. gparted to do this. Gparted still wraps e2fsck (in the case of checking a partition), but the graphical interface makes it a lot easier to select the proper hard-disk, as well as ensuring that you run these tools with the correct arguments. With the wrong arguments you risk severe loss of data!
If it helps anyone, I had a similar issue (rsync/rsnapshot backups, on an affected file). I posted my problem/solution, here:
rsnapshot (rsync) backup error on arch linux x86_64 system; a corrupted, deeply-nested file was throwing that error, also shown when I tried to delete that file:
sudo rm -fR hourly.5/ rm: cannot remove 'hourly.5/snapshot_root/mnt/Vancouver/temp/temp - old/temp - 09 (Dec 07, 2014 - Sep 02, 2015)/a_OLD-gmail/victoria.a.stuart@gmail.com/[Gmail]/LINUX/rsync, rsnapshot; Other backups/19.bak': Structure needs cleaning cd mnt/Vancouver/temp/temp\ -\ old/temp\ -\ 09\ \(Dec\ 07\,\ 2014\ -\ Sep\ 02\,\ 2015\)/a_OLD-gmail/victoria.a.stuart@gmail.com/\[Gmail\]/LINUX/rsync\,\ rsnapshot\;\ Other\ backups/ ls -l ls: cannot access '19.bak': Structure needs cleaning total 0 -. ? ? ? ? ? 19.bak ## [ See also: https://www.reddit.com/r/linuxquestions/comments/4b47r2/has_anyone_ever_gotten_structure_needs_cleaning/ ]
My backup drive is /dev/sda1.
sudo umount /dev/sda1 sudo fsck.ext4 /dev/sda1 ## - Edit (2023-02): a precautionary note: you cannot run fsck on NAS - network associated (RAID) hard disk drives - commonly sold online (Amazon; . ), and which look like regular HDD; repairs to NAS drives require special software. If you apply fsck to a NAS HDD, you'll likely corrupt it.
Rebooted: all seems fine. Went into backups drive, deleted that problematic file:
/mnt/Backups/rsnapshot_backups/hourly.5/snapshot_root/mnt/Vancouver/temp/temp - old/temp - 09 (Dec 07, 2014 - Sep 02, 2015)/a_OLD-gmail/victoria.a.stuart@gmail.com/[Gmail]/LINUX/rsync, rsnapshot; Other backups/19.bak [Update: yes; that worked: my backups are running normally, again! 🙂 ]
This just got upvoted, prompting me to peek at this older answer. Note: don't run fsck on a NAS (Network Attached Storage) hard drive - commonly sold online (Amazon; . ) particularly one that is used as a regular (non-RAID) drive - you will corrupt it. If you have a non-RAID system, buy non-NAS HDD, and routinely back up to a second non-NAS HDD. When a NAS drive fails, it's probably best to just toss it, as (from what I understand) specialized software and procedures are needed to recover lost data, from the RAID (for which NAS drives are intended).
File systems sometimes are not in order and need cleaning. This can be done by fsck command. But remember, you must run fsck only to unmounted partitions to avoid risk of file corruption.
If your file system is ext4, try running this command :
a list of what options do is always welcome, as we should teach people to understand what they are running instead of just copy/paste some magic command. So -A is check all fstab devices, -R will exclude the root filesystem from -A, as it is mounted readwrite and usually can't be check at this time (can be check offline or if mounted read-only) , -t ext4 to limit the list to ext4 filesystems (ignoring the others that may exist) and -y to auto-answer yes to the repair questions
Not that I'd run a command like that without looking up the CLOs first, but I do appreciate the heads up. That's a bad answer in that it's unlikely -A ll the drives/partitions attached to a system will have issues. As OP said, they're using an external HDD which means they'd want to scan that specific drive/partition, not all minus root.
I fixed this problem with this command in my proxmox shell:
Now backup and all is good!
So your answer is — stop any software that might be using the disk, run fsck , and resume using the disk. How does this differ from DepressedDaniel's answer?
Thjis is very different because it is a special case of proxmox containers. You can't fsck inside the lxc container nor have (a easy) direct control of the lxc devices to run the fsck, so this commands are the correct and easier answer for errors in a proxmox lxc container
@Melebius you are correct for the question but i was referring to this specific answer. While this answer do not solves the original question, it is a useful answer for the same problem in a proxmox container. It is useful for people that arrive here via search (like me)
I got the same error message from rsync and same error message from rm when I tried to delete the file. Because the filesystem was the root filesystem there was no way to use fsck. But when I just rebooted the system the file was gone and the backup succeeded. I have no clue why this worked but at least that's an easy fix and it's worth to try a reboot first.
Just recently had this problem myself, so I'll add what I did here.
Ubuntu 20.04 3 disks (4tb each) - (sdd,sde and sdf) formatted with raid5 as md0. MD0/raid was formatted with LUKS. Luks contained LVM with 1 partition of ext4 mounted as /home.
sdd 8:48 0 3.7T 0 disk └─md0 9:0 0 7.3T 0 raid5 └─vgraid-lvraid 253:0 0 7.3T 0 lvm └─luks-raid 253:4 0 7.3T 0 crypt /home sde 8:64 0 3.7T 0 disk └─md0 9:0 0 7.3T 0 raid5 └─vgraid-lvraid 253:0 0 7.3T 0 lvm └─luks-raid 253:4 0 7.3T 0 crypt /home sdf 8:80 0 3.7T 0 disk └─md0 9:0 0 7.3T 0 raid5 └─vgraid-lvraid 253:0 0 7.3T 0 lvm └─luks-raid 253:4 0 7.3T 0 crypt /home No matter what I tried, couldn't un-mount /home.
I ended up rebooting into a Live CD (Ubuntu 20.04), and used the 'Try before installing' option.
Mounted/activated md0 raid using
Then started up gnome-disks
Using Disks, I mounted the LUKS volume, and this gave access to the EXT4 partition where I used gnome-disks to perform filesystem check and repair on the unmounted ext4 partition.
Rebooted back to normal, and all was fixed.
It's usual error when try remove .Trash-0 if you try remove windows in encoding CP1251 file system on Linux system. So fs is corrupt but this dosn't metter. Fs of Windows always corrupt as see from Linux. But is not so. You may try open this file from windows os All will be well. And then remove it in windows. And only after that clean trash.
It's not a solution: it's LUKS drive encrypted with LUKS. Accessing (or at least trying access) it from Windows is in my opinion not the best idea.
Contrary to others, I faced this issue, not because any of my disks were failing, but my lower RAM frequencies would cause weird issues including this one while under heavy load.
Basically, for me, it was my DDR4 RAM being set to 2133mhz instead of 3000mhz, coupled with 12700K, and a 20 thread full IO workload that resulted in me facing this error repeatedly. No amount of memtest,Aida64 stress test, etc would reveal this issue, until thanks to God, I finally meddled with ram frequency and it all got sorted out and errors were no more!
This seems like an issue especially for Intel's 12th gen cpus/motherboard that use DDR4 instead of DDR5, I'm not sure if this can happen in other cases, but if you are on the 12th gen, and use DDR4 and face such weird error, definitely check your RAM frequencies.
You must log in to answer this question.
Linked
Related
Hot Network Questions
Subscribe to RSS
To subscribe to this RSS feed, copy and paste this URL into your RSS reader.
Site design / logo © 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA . rev 2023.7.13.43531
Linux is a registered trademark of Linus Torvalds. UNIX is a registered trademark of The Open Group.
This site is not affiliated with Linus Torvalds or The Open Group in any way.
By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.
lsblk: Structure needs cleaning
That's not a command to free up space – that's a command to destroy all data on the 'sda1' partition. So while it's technically true that you now have 8 GiB of free space, it also means you have nothing else left on the system.
but while doing so, the command ironically ran out of space.
That message is expected here, as you're writing an infinite-length source (/dev/zero which keeps producing data forever) to a fixed-size destination (the /dev/sda1 partition which is exactly 8 GiB in size).
You get different error messages depending on which data happened to survive in the RAM cache kept by the OS, vs which data needs to be re-read from disk. Very recently used commands are still cached in RAM and will fail differently than those which haven't been cached yet. For example:
- The OS can't even find the file, due to filesystem information having been erased. You get "File not found" (ENOENT).
- The OS can still find the file based on cached directory information in RAM, but cannot read the file's own metadata (extent list) as it has been erased. You get "Structure needs cleaning" (EUCLEAN/EFSCORRUPTED).
- The OS can open and read the beginning of the program which is still intact in RAM, but the remainder has to be loaded from disk, which either fails or returns garbage data due to having been erased. You get various crashes.
Sources recommend to run fsck but I cannot because my system does not support the command:
It probably did up until you erased everything.
However, in the case where the / filesystem gets legitimately corrupted, the usual recommendation is to run fsck from another system (i.e. from a rescue liveCD), which will usually have all the necessary tools whether the original system had them or not. It'll also have greater flexibility in actually repairing the filesystem, since repairs often cannot be done while the filesystem is still mounted.
What is the recommended curse of action?
The recommended curse is [omitted].
The recommended course of action is to restore the system from backups, or reinstall it.