Repository navigation
[tests] Add a native Android MSTest source-generation proof - #13432
Closed
simonrozsival wants to merge 3 commits into
Closed
simonrozsival wants to merge 3 commits into
simonrozsival wants to merge 3 commits into
Conversation
Context: 716855e Add an isolated native MTP/MSTest app and two referenced test assemblies exercising generated invocation, inherited lifecycle/context, duplicate data and DynamicData rows, async tests, JNI, attachments, and intentional diagnostics. Explicit Publish=true retains private test runtime packages in the Android deployment closure. Verify identical 20-row native identity/outcome multisets in Debug, full-trim CoreCLR and actual NativeAOT, including rooted/rootless fixture comparisons. The default project-mode MTP selection passes 15 tests with 2 skips. Keep existing NUnit/JNI/resource roots untouched; generated descriptor completeness remains false, and standalone listing is outside this proven contract. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve the accepted isolated Android proof on the reconciled host/CI foundation without migrating the production runtime test applications in this layer. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
Closing: this standalone proof project is not needed. The Android MSTest migration is folded into #13434. |
This was referenced Nov 11, 2026
simonrozsival
added a commit
that referenced
this pull request
Nov 11, 2026
Context: https://learn.microsoft.com/dotnet/core/testing/unit-testing-with-dotnet-test Context: https://github.com/microsoft/testfx/blob/88a31ec394abe9f5c5dcf596f0a150ff998d3a0e/docs/source-generator/design.md Context: #13432 Context: #13433 The on-device runtime tests (`Mono.Android.NET-Tests`, its `Java.Interop-Tests` fixtures, and `Xamarin.Android.JcwGen-Tests`) run NUnit through `TestRunner.Core`. NUnit discovers and invokes tests via reflection, which fights trimming and NativeAOT. To keep it working, the test app roots entire assemblies (`<TrimmerRootAssembly RootMode="All"/>` for `Mono.Android.NET-Tests` and `Java.Interop-Tests`), sets `IlcGenerateCompleteTypeMetadata=true`, and ships a `NativeAOT.rd.xml` whose only purpose is to keep NUnit's own reflection alive: * `EquatablesComparer.GetEquatableImplementations()` calls `GetGenericArguments()[0]` on closed `IEquatable<T>` instantiations whose metadata ILC otherwise trims, and * `TaskAwaitAdapter+GenericAdapter<VoidTaskResult>` is built via `Type.MakeGenericType()` for every `async Task` test. That is, we were testing a trimmed app whose trimming had been substantially turned off for the benefit of the test framework. Migrate these suites to MSTest running natively on Microsoft.Testing.Platform (MTP), with `MSTest.SourceGeneration` emitting the test registry and invocation delegates at compile time. ### Android runtime tests * Add `tests/TestRunner.MSTest.Android`, an Android `Instrumentation` that hosts MTP in-process, registers the fixture assemblies via `AddMSTest`, applies `IncludeCategories`/`ExcludeCategories` from `runtimeconfig.json` (and `-e include`, `-e exclude`, `-e noexclusions true` extras), and streams results back through the existing `dotnet test` Android protocol. Test failures return native exit code `2`; infrastructure failures cancel instrumentation instead of reporting an empty successful run. * Convert test attributes/assertions to MSTest, preserving each case's canonical name, data rows, categories and multiplicity. Exception assertions keep their original exactness; where NUnit's `Assert.Throws` semantics differed, the test body was restructured rather than weakened. * Remove `NativeAOT.rd.xml`, `IlcGenerateCompleteTypeMetadata`, and the whole-assembly roots for `Mono.Android.NET-Tests` and `Java.Interop-Tests`. * Keep roots that protect product/JNI behavior rather than test discovery: `Mono.Android-Test.Library.NET`, `StartupHook`, converter/reflection roots, the JCW whole-app root, and new narrow `TrimmerRoots.xml` entries for `RegisterMeOnNewNativeThreadOne` (loaded only by Java name from native code; without it `RegisterTypeOnNewNativeThread` returns `-104`) and the three `ExportField` initializers on `ExportStaticField`/`ExportInstanceField` (called only from Java field initialization; without them the generated Java `ANSWER` field disappears). The existing `apk-instrumentation.yaml` CI path is unchanged: it still runs `dotnet test` from the app directory and passes `IncludeCategories=JniReferenceLeak` for the dedicated JNI lanes. ### SourceWriter host tests `Xamarin.SourceWriter-Tests` moves to MSTest/MTP as the host-side pilot. A new opt-in `runNativeMtpHostTests` pipeline parameter (default `false`) adds an independent Linux stage that installs `$(DotNetSdkVersion)` and runs: DOTNET_TEST_RUNNER=Microsoft.Testing.Platform dotnet test \ --project tests/Xamarin.SourceWriter-Tests/Xamarin.SourceWriter-Tests.csproj \ -c Debug --report-trx --results-directory $(Agent.TempDirectory)/TestResults The TRX is published with `PublishTestResults@2` and `failTaskOnMissingResultsFile: true`. MTP selection is scoped to that one step via `DOTNET_TEST_RUNNER`, so every other NUnit suite keeps using VSTest and all existing stages are unchanged; `skipTests` still skips the new stage. An earlier revision drove this stage through a custom Python adapter (`run-native-mtp-tests.py`) that did UID-based slicing, retried failed tests and rewrote TRX files. For an 11-test suite that was all cost and no benefit, so it was removed in favor of the plain `dotnet test` invocation above: MTP's exit code already fails the step, and the standard Azure task publishes the results. ### Package versions All migrated suites take their versions from `eng/Versions.props`: the Maestro-flowed `$(MSTestPackageVersion)` (`4.6.0-preview.26531.5`) plus a new testfx dependency in `eng/Version.Details.xml` for `$(MicrosoftTestingExtensionsTrxReportPackageVersion)`, so TrxReport flows in lockstep with MSTest. `Microsoft.Testing.Platform.MSBuild` comes transitively from `MSTest.TestAdapter`. The repo-wide `$(MicrosoftTestingPlatformVersion)` used by `Microsoft.Android.Run` is not touched. ### Documentation * `tests/Xamarin.SourceWriter-Tests/README.md` describes how to build, run and filter the MTP suite. * `Documentation/workflow/UnitTests.md` and the `tests` skill catalog describe the MSTest suites and their `dotnet test` commands. ### Results Release, full trim, API-35 arm64 emulator, local SDK built from this branch (`make prepare leeroy CONFIGURATION=Release`), MSTest `4.6.0-preview.26531.5`. NUnit baseline is a fresh run of the original suites on the same `main` (`598780a6`): | Suite | Cases | Passed | Skipped | NUnit baseline | | -------------- | ----: | -----: | ------: | -------------- | | Mono CoreCLR | 1046 | 1038 | 8 | identical | | Mono NativeAOT | 983 | 975 | 8 | identical | | JCW CoreCLR | 47 | 46 | 1 | identical | Each was run from its test project directory after `./dotnet-local.sh build -t:Install -c Release [-p:PublishAot=true] <project>`, via `../../../dotnet-local.sh test <project> --no-build -c Release [-p:PublishAot=true] --report-trx`. Identities, categories and multiplicities were also compared case by case earlier in development, along with the `JniReferenceLeak` (Mono 6, JCW 1) and NTLM opt-in lanes; those category lanes were not re-run on this head. `Xamarin.SourceWriter-Tests`: 11/11 passed with SDK `11.0.100-rc.3.26522.148` (the pipeline's `$(DotNetSdkVersion)`) and SDK 12; a filter matching nothing exits `8`, so failures propagate. On-device execution time was measured but is noisy: median 16.9s (MSTest) vs. 19.2s (NUnit), while repeated runs of the *unchanged* NUnit app ranged 16.7s–43.4s. This PR does not claim a CI speedup. ### Known issues * JCW under NativeAOT aborts with `SIGABRT` (`RhFailFast`) once `JavaAbstractMethodTest` starts, under **both** the original NUnit and the migrated MSTest app. This is pre-existing and not addressed here. * `MSTest.SourceGeneration` still reports incomplete descriptors for some fixtures (generic helpers, inherited members) and retains a bounded reflection fallback; this is not "reflection-free" test execution. * Only API-35 arm64 with CoreCLR and NativeAOT was exercised; the Azure stages have not been run against this revision. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an isolated three-assembly Android MSTest/MTP proof before migrating the
runtime suites. Cover generated metadata, inherited lifecycle/context, async
tests, duplicate data rows, JNI, physical attachments and deliberate failure
diagnostics with UID-based streaming/TRX verification.
The original isolated proof's rooted/rootless CoreCLR/NativeAOT and project-mode
results are preserved historical selected-head/SDK evidence. They are not
silently promoted to the newer SDK. Additional final-integration evidence below
uses the production runner from the migration layer through an artifact-only
instrumentation overlay; it does not claim an independent new-SDK rerun of this
lower layer's original instrumentation.
Recorded commands use
$SESSION_FILESfor the preserved validation bundle andowned API35 arm64/ADB5639/emulator5636, normal Release/full trimming, and private
SDK/runtime
26551.101:K_JCW_PROOF_COMPILER_ADMITTED=yes bash "$SESSION_FILES/build-final-jcw-proof-coreclr.sh"andK_JCW_PROOF_QUIET_ADMITTED=yes bash "$SESSION_FILES/run-final-jcw-proof-coreclr.sh":passed; shared CoreCLR proof has exactly 20 rows:
15 passed, two skipped and three intentional failures/native exit 2.
Listing/empty selection explicitly cancel; arbitrary project filtering rejects
rather than silently selecting cases. These negatives are CoreCLR evidence only.
K_SHARED_PROOF_AOT_COMPILER_ADMITTED=yes bash "$SESSION_FILES/build-final-shared-proof-nativeaot.sh":first compiler attempt failed with an unknown
cscCLR assertion;the identical compiler retry built/linked/installed successfully
(10 warnings, zero errors). Its separate artifact-copy guard failed on a wrong
debug-image directory; corrected copying resumed without rebuilding.
Both failures remain retained. Actual installed APK is NativeAOT/no CoreCLR;
only explicit ILC root is Resource.Designer, with no RD/descriptor/complete
metadata option or added roots/suppressions/GC flags.
K_SHARED_PROOF_AOT_QUIET_ADMITTED=yes bash "$SESSION_FILES/run-final-shared-proof-nativeaot.sh":passed, exact same 20 UID/metadata/outcome rows as current CoreCLR,
expected native exit 2, all three registrations, exactly-once lifecycle,
duplicate row values, physical attachment bytes/TRX links and generated
invocation/failure diagnostics. Compiler/pre/post APK-set hashes agree.
Parent offline review additionally reconciled all TRX result/definition/entry
UID multisets and execution-ID pairs, full failure messages/stacks against
streaming, and the exact execution-specific attachment path and bytes.
Fixture descriptors retain bounded fallback where incomplete. Empty native
per-node assembly metadata is not fabricated into independent defining-assembly
evidence. No universal zero-reflection, x86_64 device, Mono runtime, or new Azure
acceptance claim is made.