From f3812461b543698feccbf357857245ea37d7b15d Mon Sep 17 00:00:00 2001 From: Leigh McCulloch <351529+leighmcculloch@users.noreply.github.com> Date: Fri, 17 Jan 2025 15:47:11 -0800 Subject: [PATCH] Update testing guide structure and add coverage testing (#1182) --- docs/build/guides/testing/code-coverage.mdx | 53 +++++++++++++++++++ .../build/guides/testing/coverage-testing.mdx | 9 ---- ...differential-tests-with-test-snapshots.mdx | 2 +- .../guides/testing/differential-tests.mdx | 2 +- docs/build/guides/testing/fuzzing.mdx | 2 +- .../integration-tests-with-mainnet-data.mdx | 3 +- .../guides/testing/integration-tests.mdx | 2 +- .../build/guides/testing/mutation-testing.mdx | 4 +- .../guides/testing/test-contract-auth.mdx | 1 + .../guides/testing/test-contract-events.mdx | 1 + 10 files changed, 63 insertions(+), 16 deletions(-) create mode 100644 docs/build/guides/testing/code-coverage.mdx delete mode 100644 docs/build/guides/testing/coverage-testing.mdx diff --git a/docs/build/guides/testing/code-coverage.mdx b/docs/build/guides/testing/code-coverage.mdx new file mode 100644 index 000000000..8a5def0c1 --- /dev/null +++ b/docs/build/guides/testing/code-coverage.mdx @@ -0,0 +1,53 @@ +--- +title: Code Coverage +hide_table_of_contents: true +description: Code coverage tools find code not tested. +sidebar_position: 10 +--- + +Measuring code coverage uses tools to identify lines of code that are and aren't executed by tests. Code coverage stats can give us an idea of how much of a contract is actually tested by its tests. + +:::tip + +Mutation testing is another form of coverage testing. See [Mutation Testing]. + +::: + +In rust projects the `cargo-llvm-cov` tool can be used to generate coverage stats, HTML reports, and lcov files that IDEs will load to display the coverage in the code editor. + +Install `cargo-llvm-cov` before proceeding with the other commands. + +``` +cargo install cargo-llvm-cov +``` + +## How to Get Coverage Stats + +Run the test subcommand that will run the tests and output the stats per file. + +``` +cargo llvm-cov test +``` + +## How to Generate a Coverage Report with Code + +Run the test subcommand that will run the tests and output a set of HTML files showing which lines of code are covered. + +``` +cargo llvm-cov test --html --open +``` + +The output of the command will indicate where the HTML file has been written. Open the file in a browser. + +## How to Generate an LCOV File for IDEs + +Run the test subcommand that will run the tests and output a single `lcov.info` file. + +``` +cargo llvm-cov test --lcov --output-path=lcov.info +``` + +Load the `lcov.info` file into your IDE using it's coverage feature. In VSCode this can be done by installing the [Coverage Gutters] extension and executing the `Coverage Gutters: Watch` command. + +[Coverage Gutters]: https://marketplace.visualstudio.com/items?itemName=ryanluker.vscode-coverage-gutters +[Mutation Testing]: mutation-testing.mdx diff --git a/docs/build/guides/testing/coverage-testing.mdx b/docs/build/guides/testing/coverage-testing.mdx deleted file mode 100644 index 0979d2849..000000000 --- a/docs/build/guides/testing/coverage-testing.mdx +++ /dev/null @@ -1,9 +0,0 @@ ---- -title: Coverage Testing -hide_table_of_contents: true -description: Coverage testing finds code not tested. -sidebar_position: 10 -draft: true ---- - -**TODO: Fill in example.** diff --git a/docs/build/guides/testing/differential-tests-with-test-snapshots.mdx b/docs/build/guides/testing/differential-tests-with-test-snapshots.mdx index 30b649ceb..61aad39bb 100644 --- a/docs/build/guides/testing/differential-tests-with-test-snapshots.mdx +++ b/docs/build/guides/testing/differential-tests-with-test-snapshots.mdx @@ -2,7 +2,7 @@ title: Differential Tests with Test Snapshots hide_table_of_contents: true description: Differential testing using automatic test snapshots. -sidebar_position: 8 +sidebar_position: 9 --- Tests are written to ensure that contracts behave today as expected, and in the future as well. Over time a contract may change and in all software development there remains the possibility of changes causing side-effects that are unexpected. Testing is one of the ways that we identify unexpected changes. diff --git a/docs/build/guides/testing/differential-tests.mdx b/docs/build/guides/testing/differential-tests.mdx index e9036b2f3..9a5674f8f 100644 --- a/docs/build/guides/testing/differential-tests.mdx +++ b/docs/build/guides/testing/differential-tests.mdx @@ -2,7 +2,7 @@ title: Differential Tests hide_table_of_contents: true description: Differential testing detects unintended changes. -sidebar_position: 7 +sidebar_position: 8 --- Differential testing is the testing of two things to discover differences in their behavior. diff --git a/docs/build/guides/testing/fuzzing.mdx b/docs/build/guides/testing/fuzzing.mdx index 6b588c996..396422b82 100644 --- a/docs/build/guides/testing/fuzzing.mdx +++ b/docs/build/guides/testing/fuzzing.mdx @@ -2,7 +2,7 @@ title: Fuzzing hide_table_of_contents: true description: Fuzzing and property testing to find unexpected behavior. -sidebar_position: 5 +sidebar_position: 7 --- Fuzzing is the process of providing random data to programs to identify unexpected behavior, such as crashes and panics. diff --git a/docs/build/guides/testing/integration-tests-with-mainnet-data.mdx b/docs/build/guides/testing/integration-tests-with-mainnet-data.mdx index a8fa9ecd3..04fb6f4e6 100644 --- a/docs/build/guides/testing/integration-tests-with-mainnet-data.mdx +++ b/docs/build/guides/testing/integration-tests-with-mainnet-data.mdx @@ -2,8 +2,7 @@ title: Integration Tests with Mainnet Data hide_table_of_contents: true description: Integration testing uses dependency contracts instead of mocks. -sidebar_position: 4 -draft: true +sidebar_position: 6 --- Testing with mainnet data is another form of [integration test], where not only mainnet contracts can be used, but also their data on mainnet. diff --git a/docs/build/guides/testing/integration-tests.mdx b/docs/build/guides/testing/integration-tests.mdx index 06003d7eb..b11d5defa 100644 --- a/docs/build/guides/testing/integration-tests.mdx +++ b/docs/build/guides/testing/integration-tests.mdx @@ -2,7 +2,7 @@ title: Integration Tests hide_table_of_contents: true description: Integration testing uses dependency contracts instead of mocks. -sidebar_position: 3 +sidebar_position: 5 --- Integration tests are tests that include the integration between components, and so test a larger scope such as other contracts. diff --git a/docs/build/guides/testing/mutation-testing.mdx b/docs/build/guides/testing/mutation-testing.mdx index 8bccad902..f731e1c0f 100644 --- a/docs/build/guides/testing/mutation-testing.mdx +++ b/docs/build/guides/testing/mutation-testing.mdx @@ -2,7 +2,7 @@ title: Mutation Testing hide_table_of_contents: true description: Mutation testing finds code not tested. -sidebar_position: 9 +sidebar_position: 11 --- Mutation testing is making changes to a program, either manually or automatically, to identify changes that can be made that don't get caught by tests. @@ -33,3 +33,5 @@ The `cargo-mutants` tool can be used to automatically and iteratively modify the Code that is identified as not covered by a test will be outputted as a `MISSED` line in the output. Diffs of each change that was attempted can be found in the `mutants.out/diff` directory. + +[code coverage]: code-coverage.mdx diff --git a/docs/build/guides/testing/test-contract-auth.mdx b/docs/build/guides/testing/test-contract-auth.mdx index 3cac5898c..054deb1b6 100644 --- a/docs/build/guides/testing/test-contract-auth.mdx +++ b/docs/build/guides/testing/test-contract-auth.mdx @@ -2,6 +2,7 @@ title: Test Authorization hide_table_of_contents: true description: Write tests that test contract authorization. +sidebar_position: 3 --- Tests can assert on the auths that are expected to occur. diff --git a/docs/build/guides/testing/test-contract-events.mdx b/docs/build/guides/testing/test-contract-events.mdx index 07f28562d..9360d1592 100644 --- a/docs/build/guides/testing/test-contract-events.mdx +++ b/docs/build/guides/testing/test-contract-events.mdx @@ -2,6 +2,7 @@ title: Test Events hide_table_of_contents: true description: Write tests that test contract events. +sidebar_position: 4 --- Tests can assert on events that are expected to be published.