On WASM tests in CI

[thomaseizinger]

In #4015 the topic of running WASM tests in CI came again. #3987 is another instance where this would be useful.

Writing tests isn't too difficult, one simply needs to add #[wasm_bindgen_test] as an attribute to a test. The problem lies in configuring all dependencies correctly, such that the test cases compile for wasm.

Unless explicitly gated with a cfg, all code will be compiled for a certain target. Thus, if we for example reference async-std anywhere in a test, cargo test --target wasm32-unknown-unknown will attempt to compile this dependency for this target (and fail).

In this discussion, I want to lay out some options on how we could tackle this.

[thomaseizinger]

Option 1: Correctly feature-gate code

The most correct approach would be to add the necessary feature-gates to the code. This could be done with

#![cfg(not(target_arch = "wasm32"))]

at the top of an integration test file for example or with

#[cfg(not(target_arch = "wasm32"))]
mod native_tests {

}

In the Cargo.toml, we would have to introduce sections such as:

[target.'cfg(not(target_arch = "wasm32"))'.dev-dependencies]
async-std = "..."

This will feature-gate dependencies to the correct target.

If we strictly follow this approach across the library, we can at the end run all tests using:

export CARGO_TARGET_WASM32_UNKNOWN_UNKNOWN_RUNNER=wasm-bindgen-test-runner
cargo test --target wasm32-unknown-unknown

Pros

• A simple CI setup
• Certainty that newly added tests are also executed

Cons

• Some boilerplate across all crates

[thomaseizinger]

Option 2: Manually list crates that have wasm tests

Given that most of our crates don't have WASM-specific tests, it is worth considering doing the feature-gating only for the specific crates that have them.

The feature-gating approach would be the same but we would have a list of crates in a CI script that get tested, something like:

- run: wasm-pack test transports/webtransport-websys
- run: wasm-pack test protocols/gossipsub
# ...

Pros

• Easier to setup

Cons

• New tests are not automatically picked up
• As more crates have WASM tests, the complexity problem of Option 1 appears

[thomaseizinger]

Option 3: Have a dedicated WASM-test crate

We could also add a dedicated crate, similar to interop-tests that contains specific wasm tests.

Pros

• No feature-gating necessary because that crate is only compiled for WASM

Cons

• Can only test public APIs of other crates
• More crates to maintain

[thomaseizinger]

To workaround the issue of not being able to test public APIs, we could extract specific bits of functionality into their own crate which again makes it easy to test that one for wasm.

For example, functionality such as https://github.com/libp2p/rust-libp2p/pull/4015/files#diff-53e9ca8d4155b629c0e2595817f86872cba4df8231a879d0158a215b61ba9d26R43-R56 would be a candidate here.

This still suffers from the same problem that we have more crates to maintain.

[thomaseizinger]

One could also ask whether functionality such as https://github.com/libp2p/rust-libp2p/pull/4015/files#diff-53e9ca8d4155b629c0e2595817f86872cba4df8231a879d0158a215b61ba9d26R43-R56 actually needs a test.