Panic when trying to remove files from _one_ of my datasets #16939
Labels
Type: Defect
Incorrect behavior (e.g. crash, hang)
Comments
|
I would start from memory check, considering it is non-ECC, to not cause more damage if it is the cause. Then make sure you have a backup, which is on your second option already. Then I would guess you should have a good chance to destroy only the specific dataset, since the ZAP it crashed on is dataset-specific, but it is difficult to be sure without knowing what's wrong about it exactly. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
System information
grub options:
nospectre_v2 split_lock_mitigate=0 zfs.spa_slop_shift=6 zfs.zfs_dmu_offset_next_sync=0 zfs.zfs_txg_timeout=20 zfs.zio_taskq_batch_pct=42 zfs.zfs_arc_max=1610612736 nvme.max_host_mem_size_mb=256 scsi_mod.use_blk_mq=1(I read somewherezfs_dmu_offset_next_sync=0supposedly decreases the chance of corruption;txg_timeoutis because the constant noise of my Seagate Skyhawk HDD was unbearable;zio_taskq_batch_pctis to avoid my PC becoming unresponsive during heavy writing to datasets withcompression=zstd-19; I limited ARC size to 1.5GiB to save RAM)I've been using this config (I mean an RT kernel) since 2024-12-15, i.e. for 23 days. It worked until today, except that waking up didn't succeed in most cases (complete lock-up). Probably nvidia's driver at fault (since they don't support RT officially)?... no idea. Lock-ups occurred also in random times but after many hours of uptime. A few times, immediately after some of successful wake-ups, I saw ZFS-related traces mentioning something
txg,unlock; but I ignored them......My pools:
zroot;all have
ashift 12, and properly aligned partitions.So, 2 days ago I was seeding torrents (using files on the HDD), and then I decided to load up an LLM (8 GB .gguf file on an unencrypted partition on the SSD; llama.cpp), and during the loading, my PC suddenly became fully unresponsive (even no SSH and alt+print+b). At that moment what was happening:
mmaping) on an SSD (poolzroot, unencrypted datasetzroot/data/m/c/rseqwithout compression) by llama.cppzroot, encrypted datasetzroot/data/home) by qBittorrent (metadata), Firefox.I found the timing of the lock-up suspicious, so today I decided to repeat the same scenario. This time the LLM did get fully loaded (in ~28 secs), but... 5 seconds later my PC became unresponsive again (it's like there is some connection to
zfs.zfs_txg_timeout=20).I rebooted my PC, selected the stock LTS kernel, then I saw:
Afterwards, I booted from a LiveCD (even earlier kernel:
Linux sysrescue 6.1.53-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 13 Sep 2023 09:32:00 +0000 x86_64 GNU/Linux), installed ZFS 2.2.7 (same latest version), mounted all pools, ranzpool scrub zroot, thenzpool status zroot:The bad dataset:
Its pool:
Then I backed up this dataset into another dataset... on the same pool... (
tar -I 'zstd -15 --long -T12' /m/el/r/home.tar.zst /home) successfully.Then I tried to use
zshunder my user, butzshtried to overwrite a file usingmv, the result is bad:ps aux:1000 40360 0.0 0.0 26600 764 pts/7 D+ 15:34 0:00 mv -f /home/arzeth/.zcompdump-sysrescue-5.9.0.1-dev.sysrescue.59 /home/arzeth/.zcompdump-sysrescue-5.9.0.1-devThis panic causes any process (even
ls) trying to use this dataset to become a zombie process.Then I decided to reboot (still using the livecd), because I wanted to know whether a panic would occur when deleting an old file.
So I tried
rma 3-year-old file (12 bytes) on this dataset. The error indmesgis similar:BTW, even after a panic occurs, I can still use other datasets in this pool (without rebooting).
I had 2 datasets inside this dataset, so I made another experiment: despite ZFS already having had a panic, I tried to move it out of
..../home/(I am not talking aboutmountpoint):zfs rename zroot/data/home{/,_}arzeth_dev, and got a very similar message indmesg. After the panic, I didn't reboot, and I ranzfs listwhich showed that the dataset renaming succeeded (even thoughzfs rename ....became a zombie process).Then I tried to rename another inner (child) dataset (without rebooting), but this time I saw no changes in
zfs list, so I've rebooted, ranzfs listagain just in case (the first renaming did indeed succeed). I haven't tried yet tozfs renamethis remaining inner dataset again.Panic does not occur when creating folders and moving files into them (in this corrupted dataset).
So what should I do?
zfs destroy zroot/data/home, but what if this irreversibly corrupts the pool due to a panic in the middle of the process?zpool destroy zroot,zpool create,zfs create,cp,cp,cp)Maybe also ban PREEMPT_RT in
configure.ac, just like recent 410287f banned unsupported kernel versions?The text was updated successfully, but these errors were encountered: