ZTS: replacement/resilver_restart_001 fails on FreeBSD 14+

[snajpa]

System information

Type	Version/Name
Distribution Name	FreeBSD
Distribution Version	14+
Kernel Version
Architecture
OpenZFS Version

Describe the problem you're observing

   11:22:35.19 NOTE: Testing with_early_restart deferred resilvers
  11:22:35.20 vfs.zfs.resilver_defer_percent: 0 -> 100
  11:22:35.20 SUCCESS: set_tunable32 RESILVER_DEFER_PERCENT 100
  11:22:35.21 cleared 64 events
  11:22:35.21 SUCCESS: zpool events -c
  11:22:35.21 vfs.zfs.resilver_min_time_ms: 3000 -> 50
  11:22:35.21 SUCCESS: set_tunable32 RESILVER_MIN_TIME_MS 50
  11:22:35.85 SUCCESS: zpool replace testpool1 /var/tmp/file-2 /var/tmp/spare-1
  11:22:35.86 vfs.zfs.scan_suspend_progress: 0 -> 1
  11:22:35.87 SUCCESS: set_tunable32 SCAN_SUSPEND_PROGRESS 1
  11:22:35.89 NOTE: expected 1 resilver start(s), found 1
  11:22:36.08 SUCCESS: zpool offline testpool1 /var/tmp/file-3
  11:22:37.17 SUCCESS: zpool sync testpool1
  11:22:38.25 SUCCESS: zpool online testpool1 /var/tmp/file-3
  11:22:39.30 SUCCESS: zpool sync testpool1
  11:22:39.31 NOTE: expected 2 resilver start(s) after offline/online, found 1
  11:22:39.31 expected 2 resilver start(s) after offline/online, found 1
  11:22:39.31 NOTE: Performing test-fail callback (/usr/local/share/zfs/zfs-tests/callbacks/zfs_dmesg.ksh)

Filing this with the expectation that I'll fix this, but need a few days to get to it. If anyone can get me zfs_dbgmsg at the time when this happens, that could save me some time as I hope I could see what's wrong from that without the need to play around with this on FreeBSD.

Describe how to reproduce the problem

Run ZTS replacement/resilver_restart_001 on FreeBSD 14 or 14.1

[haowuge]

I am unable to run the script. How should I proceed?
Thank you

git clone https://github.com/openzfs/zfs
./autogen.sh 
MAKE=gmake ./configure
sudo gmake install

root@LoongBuild:~ # /usr/local/share/zfs/zfs-tests/tests/functional/replacement/resilver_restart_001.ksh
/usr/local/share/zfs/zfs-tests/tests/functional/replacement/resilver_restart_001.ksh[22]: .: /include/libtest.shlib: not found
root@LoongBuild:~ # freebsd-version -ku
14.2-RC1
14.2-RC1
root@LoongBuild:~ # 

[snajpa]

@haowuge there's a helper script that wraps the testsuite script runner - ./scripts/zfs-tests.sh, takes -h for help

you can run this one test like this:

cd zfs
./scripts/zfs-tests.sh -t tests/functional/replacement/resilver_restart_001.ksh

(just please be sure to only run it on a dev machine with no data you can't afford to lose)

[haowuge]

@haowuge there's a helper script that wraps the testsuite script runner - ./scripts/zfs-tests.sh, takes -h for help

you can run this one test like this:

cd zfs
./scripts/zfs-tests.sh -t tests/functional/replacement/resilver_restart_001.ksh

(just please be sure to only run it on a dev machine with no data you can't afford to lose)

I saw: “SKIP replacement/resilver_restart_001.ksh (expected PASS)”
Did it fail to run?
I might need to wait until I get home tonight to test it; I'm afraid that remotely restarting the NAS host will result in a failure.
The currently running ZFS in the system is from 14.2-RC1, and the test script is from OpenZFS master.

[haowuge]

Is this the normal execution completed?

❯ sudo ./zfs-tests.sh  -t tests/functional/replacement/resilver_restart_001.ksh
zfs-tests.sh: This script must not be run as root.
❯ ./zfs-tests.sh  -t tests/functional/replacement/resilver_restart_001.ksh
Test: /usr/local/share/zfs/zfs-tests/tests/functional/replacement/setup (run as root) [00:00] [FAIL]
Test: /usr/local/share/zfs/zfs-tests/tests/functional/replacement/resilver_restart_001.ksh (run as root) [00:00] [SKIP]
Test: /usr/local/share/zfs/zfs-tests/tests/functional/replacement/cleanup (run as root) [00:00] [PASS]

Results Summary
PASS       1
FAIL       1
SKIP       1

Running Time:   00:00:00
Percent passed: 33.3%
Log directory:  /var/tmp/test_results/20241201T190752

Tests with results other than PASS that are expected:

Tests with result of PASS that are unexpected:

Tests with results other than PASS that are unexpected:
    SKIP replacement/resilver_restart_001.ksh (expected PASS)
    FAIL replacement/setup (expected PASS)

[snajpa]

@haowuge there's a [FAIL] at setup step, so it skipped the test; you can find the logs of the last test run in /var/tmp/test_results/current by default

[haowuge]

@haowuge there's a [FAIL] at setup step, so it skipped the test; you can find the logs of the last test run in /var/tmp/test_results/current by default

I feel like it didn't successfully complete the task😄

https://haowuge.github.io/pull_requests_images/2024-12-01/test_results-20241201T101933.tar.gz

[mcmilk]

@haowuge - you have loaded the zfs.ko module. I think you need to blacklist this one, and insert openzfs.ko.
So you are testing the shipped default zfs module - which seems not to work.