BEEFY crash-loops trying to fetch the parent of genesis

[dmitry-markin]

During versi burn-in of #4874, rev. 8f737b3, approximately half of the nodes went to the crash-loop with the following messages:

2024-06-25 14:07:14.158  INFO tokio-runtime-worker beefy: 🥩 BEEFY gadget waiting for BEEFY pallet to become available...
2024-06-25 14:07:14.706  INFO tokio-runtime-worker beefy: 🥩 BEEFY pallet available: block 71541 beefy genesis 1
2024-06-25 14:07:15.375  WARN tokio-runtime-worker beefy: header 0xb418…181f is Genesis, there is no parent for it
2024-06-25 14:07:15.375 ERROR tokio-runtime-worker beefy: 🥩 Error: Backend("Backend: header 0xb418…181f is Genesis, there is no parent for it"). Terminating.
2024-06-25 14:07:15.375 ERROR tokio-runtime-worker sc_service::task_manager: Essential task `beefy-gadget` failed. Shutting down service.    
Error: 
   0: Other: Essential task failed.

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 5 frames hidden ⋮                               
   6: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual::h8d15be7044cd74d5
      at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/result.rs:1959
   7: polkadot::main::hae29386f4b250e21
      at /builds/parity/mirrors/polkadot-sdk/polkadot/src/main.rs:31
   8: core::ops::function::FnOnce::call_once::h033b09deefa99362
      at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/ops/function.rs:250
                                ⋮ 14 frames hidden ⋮                              

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Run with RUST_BACKTRACE=full to include source snippets.

The nodes were run with the following flags:

--in-peers=100
--out-peers=100
--sync=warp
--allow-private-ipv4
--discover-local
--insecure-validator-i-know-what-i-do
--detailed-log-output
--network-backend litep2p

CC @serban300

[dmitry-markin]

The flag --sync=warp might not be relevant, as earlier in the logs there is a message

2024-06-25 14:07:14.144 ERROR main sync: Can't use warp sync mode with a partially synced database. Reverting to full sync mode.

[acatangiu]

can you do a RUST_LOG=beefy=debug run for more logs?

[dmitry-markin]

Sure, but I can't reproduce it anymore after re-deploying the nodes. Will do if the issue manifests again.

[acatangiu]

Closing for now, let's reopen if we see it again.