Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update testing guide with code coverage for fuzzing #1183

Merged
merged 6 commits into from
Jan 19, 2025
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions docs/build/guides/testing/code-coverage.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
title: Code Coverage
hide_table_of_contents: true
description: Code coverage tools find code not tested.
sidebar_position: 10
---

Measuring code coverage uses tools to identify lines of code that are and aren't executed by tests. Code coverage stats can give us an idea of how much of a contract is actually tested by its tests.

:::tip

Mutation testing is another form of coverage testing. See [Mutation Testing].

:::

In rust projects the `cargo-llvm-cov` tool can be used to generate coverage stats, HTML reports, and lcov files that IDEs will load to display the coverage in the code editor.

Install `cargo-llvm-cov` before proceeding with the other commands.

```
cargo install cargo-llvm-cov
```

## How to Get Coverage Stats

Run the test subcommand that will run the tests and output the stats per file.

```
cargo llvm-cov test
```

## How to Generate a Coverage Report with Code

Run the test subcommand that will run the tests and output a set of HTML files showing which lines of code are covered.

```
cargo llvm-cov test --html --open
```

The output of the command will indicate where the HTML file has been written. Open the file in a browser.

## How to Generate an LCOV File for IDEs

Run the test subcommand that will run the tests and output a single `lcov.info` file.

```
cargo llvm-cov test --lcov --output-path=lcov.info
```

Load the `lcov.info` file into your IDE using it's coverage feature. In VSCode this can be done by installing the [Coverage Gutters] extension and executing the `Coverage Gutters: Watch` command.

:::info

Measuring code coverage in fuzz tests requires different tooling. See [Fuzzing].

:::

[Coverage Gutters]: https://marketplace.visualstudio.com/items?itemName=ryanluker.vscode-coverage-gutters
[Mutation Testing]: mutation-testing.mdx
[Fuzzing]: fuzzing.mdx
9 changes: 0 additions & 9 deletions docs/build/guides/testing/coverage-testing.mdx

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Differential Tests with Test Snapshots
hide_table_of_contents: true
description: Differential testing using automatic test snapshots.
sidebar_position: 8
sidebar_position: 9
---

Tests are written to ensure that contracts behave today as expected, and in the future as well. Over time a contract may change and in all software development there remains the possibility of changes causing side-effects that are unexpected. Testing is one of the ways that we identify unexpected changes.
Expand Down
2 changes: 1 addition & 1 deletion docs/build/guides/testing/differential-tests.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Differential Tests
hide_table_of_contents: true
description: Differential testing detects unintended changes.
sidebar_position: 7
sidebar_position: 8
---

Differential testing is the testing of two things to discover differences in their behavior.
Expand Down
45 changes: 44 additions & 1 deletion docs/build/guides/testing/fuzzing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Fuzzing
hide_table_of_contents: true
description: Fuzzing and property testing to find unexpected behavior.
sidebar_position: 5
sidebar_position: 7
---

Fuzzing is the process of providing random data to programs to identify unexpected behavior, such as crashes and panics.
Expand Down Expand Up @@ -116,6 +116,47 @@ There is another tool for fuzzing Rust code, `cargo-afl`. See the [Rust Fuzz boo

:::

### How to Get Code Coverage of Fuzz Tests

Getting code coverage data for fuzz tests requires some different tooling than when doing the same for regular Rust tests.

1. Run the fuzz tests until it has produced a corpus, just as in step 7 above.

```
cargo +nightly fuzz run --sanitizer thread fuzz_target_1
```

2. Install the llvm-tools for the nightly compiler.

```
rustup component add --toolchain nightly llvm-tools-preview
```

3. Run the fuzz coverage command that'll execute the corpus and write coverage data to the coverage directory in the `profdata` format.

```
cargo +nightly fuzz coverage --sanitizer thread fuzz_target_1
```

4. Run the llvm-cov command to convert the profdata file to an lcov file.

```
LLVM_TOOLS_PATH=$(dirname $(find $(rustc +nightly --print sysroot) -name 'llvm-cov'))
$LLVM_TOOLS_PATH/llvm-cov export -instr-profile=fuzz/coverage/fuzz_target_1/coverage.profdata \
-format=lcov \
-object target/aarch64-apple-darwin/coverage/aarch64-apple-darwin/release/fuzz_target_1 \
--ignore-filename-regex "rustc" > lcov.info
```

Load the `lcov.info` file into your IDE using it's coverage feature. In VSCode this can be done by installing the [Coverage Gutters] extension and executing the `Coverage Gutters: Watch` command.

:::tip

To measure code coverage of regular Rust tests, see [Code Coverage].

:::


[increment example]: https://github.com/stellar/soroban-examples/blob/main/increment/src/lib.rs
[Differential Testing with Test Snapshots]: ./differential-tests-with-test-snapshots.mdx
[stellar contract fetch]: ../../../tools/developer-tools/cli/stellar-cli.mdx#stellar-contract-fetch
Expand All @@ -124,3 +165,5 @@ There is another tool for fuzzing Rust code, `cargo-afl`. See the [Rust Fuzz boo
[stellar/rs-soroban-sdk#1360]: https://github.com/stellar/rs-soroban-sdk/issues/1360
[fuzzing example]: ../../smart-contracts/example-contracts/fuzzing.mdx
[Rust Fuzz Book]: https://rust-fuzz.github.io/book
[Code Coverage]: code-coverage.mdx
[Coverage Gutters]: https://marketplace.visualstudio.com/items?itemName=ryanluker.vscode-coverage-gutters
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,7 @@
title: Integration Tests with Mainnet Data
hide_table_of_contents: true
description: Integration testing uses dependency contracts instead of mocks.
sidebar_position: 4
draft: true
sidebar_position: 6
---

Testing with mainnet data is another form of [integration test], where not only mainnet contracts can be used, but also their data on mainnet.
Expand Down
2 changes: 1 addition & 1 deletion docs/build/guides/testing/integration-tests.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Integration Tests
hide_table_of_contents: true
description: Integration testing uses dependency contracts instead of mocks.
sidebar_position: 3
sidebar_position: 5
---

Integration tests are tests that include the integration between components, and so test a larger scope such as other contracts.
Expand Down
4 changes: 3 additions & 1 deletion docs/build/guides/testing/mutation-testing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Mutation Testing
hide_table_of_contents: true
description: Mutation testing finds code not tested.
sidebar_position: 9
sidebar_position: 11
---

Mutation testing is making changes to a program, either manually or automatically, to identify changes that can be made that don't get caught by tests.
Expand Down Expand Up @@ -33,3 +33,5 @@ The `cargo-mutants` tool can be used to automatically and iteratively modify the
Code that is identified as not covered by a test will be outputted as a `MISSED` line in the output.

Diffs of each change that was attempted can be found in the `mutants.out/diff` directory.

[code coverage]: code-coverage.mdx
1 change: 1 addition & 0 deletions docs/build/guides/testing/test-contract-auth.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
title: Test Authorization
hide_table_of_contents: true
description: Write tests that test contract authorization.
sidebar_position: 3
---

Tests can assert on the auths that are expected to occur.
Expand Down
1 change: 1 addition & 0 deletions docs/build/guides/testing/test-contract-events.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
title: Test Events
hide_table_of_contents: true
description: Write tests that test contract events.
sidebar_position: 4
---

Tests can assert on events that are expected to be published.
Expand Down