Repository navigation
[tests] Migrate Android runtime and JCW suites to native MSTest - #13433
Closed
simonrozsival wants to merge 11 commits into
Closed
simonrozsival wants to merge 11 commits into
simonrozsival wants to merge 11 commits into
Conversation
Register fixture assemblies with native MSTest, export source-generated identity evidence and TRX results, and preserve the Android streaming protocol and category policy without NUnit compatibility paths. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve data multiplicity, inherited lifecycle, explicit JNI opt-in, diagnostics, repeat behavior, and original canonical case mapping. Remove only proven Mono fixture/NUnit roots, retain the native-thread Java root and independent behavioral roots, and retain the JCW whole-app root for its pre-existing NativeAOT NewMethod abort. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use ThrowsExactly for the three remaining NUnit Throws assertions so the migration does not silently accept derived exceptions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Merge the main598780a62f-based host/CI and Android proof layers. Preserve upstream debugger-boundary categories, handled-exception ABI defaults, policy-once assertions, original exception identity and pending-state checks while using the migrated MSTest runner. Retain all exact exception assertion corrections and original accepted histories. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the three ExportField helper methods with method-only JNI roots after removing NUnit discovery roots. The linked app otherwise loses the primitive static initializer and generated ANSWER field. Preserve the existing strict field checks and all test/assembly discovery behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use an extern alias for the JNI fixture assembly, keep nullable class-name validation explicit, and correct expected/actual argument presentation. Preserve original test names, categories, boxed-reference and vacuous value-type checks. Document remaining generated trimming contracts and fresh original runtime baselines without suppressing diagnostics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The app's JavaAs case constructs the JNI fixture assembly's MyJavaInterfaceImpl in addition to selecting its assembly marker. Preserve that exact peer type through the same extern alias; no test identity or assertion changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Link the dependent Android migration documentation to the SourceWriter instructions and replace the removed custom slicing description with direct MTP execution. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
simonrozsival
added a commit
that referenced
this pull request
Nov 11, 2026
Context: https://learn.microsoft.com/dotnet/core/testing/unit-testing-with-dotnet-test Context: https://github.com/microsoft/testfx/blob/88a31ec394abe9f5c5dcf596f0a150ff998d3a0e/docs/source-generator/design.md Context: #13432 Context: #13433 The on-device runtime tests (`Mono.Android.NET-Tests`, its `Java.Interop-Tests` fixtures, and `Xamarin.Android.JcwGen-Tests`) run NUnit through `TestRunner.Core`. NUnit discovers and invokes tests via reflection, which fights trimming and NativeAOT. To keep it working, the test app roots entire assemblies (`<TrimmerRootAssembly RootMode="All"/>` for `Mono.Android.NET-Tests` and `Java.Interop-Tests`), sets `IlcGenerateCompleteTypeMetadata=true`, and ships a `NativeAOT.rd.xml` whose only purpose is to keep NUnit's own reflection alive: * `EquatablesComparer.GetEquatableImplementations()` calls `GetGenericArguments()[0]` on closed `IEquatable<T>` instantiations whose metadata ILC otherwise trims, and * `TaskAwaitAdapter+GenericAdapter<VoidTaskResult>` is built via `Type.MakeGenericType()` for every `async Task` test. That is, we were testing a trimmed app whose trimming had been substantially turned off for the benefit of the test framework. Migrate these suites to MSTest running natively on Microsoft.Testing.Platform (MTP), with `MSTest.SourceGeneration` emitting the test registry and invocation delegates at compile time. ### Android runtime tests * Add `tests/TestRunner.MSTest.Android`, an Android `Instrumentation` that hosts MTP in-process, registers the fixture assemblies via `AddMSTest`, applies `IncludeCategories`/`ExcludeCategories` from `runtimeconfig.json` (and `-e include`, `-e exclude`, `-e noexclusions true` extras), and streams results back through the existing `dotnet test` Android protocol. Test failures return native exit code `2`; infrastructure failures cancel instrumentation instead of reporting an empty successful run. * Convert test attributes/assertions to MSTest, preserving each case's canonical name, data rows, categories and multiplicity. Exception assertions keep their original exactness; where NUnit's `Assert.Throws` semantics differed, the test body was restructured rather than weakened. * Remove `NativeAOT.rd.xml`, `IlcGenerateCompleteTypeMetadata`, and the whole-assembly roots for `Mono.Android.NET-Tests` and `Java.Interop-Tests`. * Keep roots that protect product/JNI behavior rather than test discovery: `Mono.Android-Test.Library.NET`, `StartupHook`, converter/reflection roots, the JCW whole-app root, and new narrow `TrimmerRoots.xml` entries for `RegisterMeOnNewNativeThreadOne` (loaded only by Java name from native code; without it `RegisterTypeOnNewNativeThread` returns `-104`) and the three `ExportField` initializers on `ExportStaticField`/`ExportInstanceField` (called only from Java field initialization; without them the generated Java `ANSWER` field disappears). The existing `apk-instrumentation.yaml` CI path is unchanged: it still runs `dotnet test` from the app directory and passes `IncludeCategories=JniReferenceLeak` for the dedicated JNI lanes. ### SourceWriter host tests `Xamarin.SourceWriter-Tests` moves to MSTest/MTP as the host-side pilot. A new opt-in `runNativeMtpHostTests` pipeline parameter (default `false`) adds an independent Linux stage that installs `$(DotNetSdkVersion)` and runs: DOTNET_TEST_RUNNER=Microsoft.Testing.Platform dotnet test \ --project tests/Xamarin.SourceWriter-Tests/Xamarin.SourceWriter-Tests.csproj \ -c Debug --report-trx --results-directory $(Agent.TempDirectory)/TestResults The TRX is published with `PublishTestResults@2` and `failTaskOnMissingResultsFile: true`. MTP selection is scoped to that one step via `DOTNET_TEST_RUNNER`, so every other NUnit suite keeps using VSTest and all existing stages are unchanged; `skipTests` still skips the new stage. An earlier revision drove this stage through a custom Python adapter (`run-native-mtp-tests.py`) that did UID-based slicing, retried failed tests and rewrote TRX files. For an 11-test suite that was all cost and no benefit, so it was removed in favor of the plain `dotnet test` invocation above: MTP's exit code already fails the step, and the standard Azure task publishes the results. ### Package versions All migrated suites take their versions from `eng/Versions.props`: the Maestro-flowed `$(MSTestPackageVersion)` (`4.6.0-preview.26531.5`) plus a new testfx dependency in `eng/Version.Details.xml` for `$(MicrosoftTestingExtensionsTrxReportPackageVersion)`, so TrxReport flows in lockstep with MSTest. `Microsoft.Testing.Platform.MSBuild` comes transitively from `MSTest.TestAdapter`. The repo-wide `$(MicrosoftTestingPlatformVersion)` used by `Microsoft.Android.Run` is not touched. ### Documentation * `tests/Xamarin.SourceWriter-Tests/README.md` describes how to build, run and filter the MTP suite. * `Documentation/workflow/UnitTests.md` and the `tests` skill catalog describe the MSTest suites and their `dotnet test` commands. ### Results Release, full trim, API-35 arm64 emulator, local SDK built from this branch (`make prepare leeroy CONFIGURATION=Release`), MSTest `4.6.0-preview.26531.5`. NUnit baseline is a fresh run of the original suites on the same `main` (`598780a6`): | Suite | Cases | Passed | Skipped | NUnit baseline | | -------------- | ----: | -----: | ------: | -------------- | | Mono CoreCLR | 1046 | 1038 | 8 | identical | | Mono NativeAOT | 983 | 975 | 8 | identical | | JCW CoreCLR | 47 | 46 | 1 | identical | Each was run from its test project directory after `./dotnet-local.sh build -t:Install -c Release [-p:PublishAot=true] <project>`, via `../../../dotnet-local.sh test <project> --no-build -c Release [-p:PublishAot=true] --report-trx`. Identities, categories and multiplicities were also compared case by case earlier in development, along with the `JniReferenceLeak` (Mono 6, JCW 1) and NTLM opt-in lanes; those category lanes were not re-run on this head. `Xamarin.SourceWriter-Tests`: 11/11 passed with SDK `11.0.100-rc.3.26522.148` (the pipeline's `$(DotNetSdkVersion)`) and SDK 12; a filter matching nothing exits `8`, so failures propagate. On-device execution time was measured but is noisy: median 16.9s (MSTest) vs. 19.2s (NUnit), while repeated runs of the *unchanged* NUnit app ranged 16.7s–43.4s. This PR does not claim a CI speedup. ### Known issues * JCW under NativeAOT aborts with `SIGABRT` (`RhFailFast`) once `JavaAbstractMethodTest` starts, under **both** the original NUnit and the migrated MSTest app. This is pre-existing and not addressed here. * `MSTest.SourceGeneration` still reports incomplete descriptors for some fixtures (generic helpers, inherited members) and retains a bounded reflection fallback; this is not "reflection-free" test execution. * Only API-35 arm64 with CoreCLR and NativeAOT was exercised; the Azure stages have not been run against this revision. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrate Mono.Android, Java.Interop and JCW fixtures to the shared native MSTest
runner while preserving canonical cases, categories, multiplicity and original
assertion behavior. Remove obsolete Mono/Java.Interop discovery roots and NUnit
metadata inputs only after their consumers move. Retain independent library,
StartupHook, JNI/converter/reflection roots and the JCW whole-app root.
Keep three narrow Java-only ExportField initializer roots and selected-main
cross-assembly JNI aliases needed by actual trimmed execution.
Final source
5f00de3fwas rebuilt/tested using private SDK/runtime26551.101,normal Release/fulltrim, API35 arm64 only.
$SESSION_FILESrefers to preservedcommands/evidence; original NUnit authority is exact selected main598, not a
newer main or an excluded-row inventory substituted for execution.
bash "$SESSION_FILES/run-final-coreclr.sh": passed, raw and independentproject both exactly 1046 rows, 1038 passed/eight skipped.
bash "$SESSION_FILES/run-final-nativeaot.sh": passed, actual NativeAOTraw and independent project both exactly 983 rows, 975 passed/eight skipped.
K_JCW_PROOF_QUIET_ADMITTED=yes bash "$SESSION_FILES/run-final-jcw-proof-coreclr.sh":passed, JCW raw/project both exactly 47 rows, 46 passed/one skipped;
shared runner and explicit negative protocol controls also verified.
K_JNI_COMPILER_ADMITTED=yes bash "$SESSION_FILES/build-final-jni-coreclr.sh"andK_JNI_QUIET_ADMITTED=yes bash "$SESSION_FILES/run-final-jni-coreclr.sh":passed, actual category runtimeconfig route; fresh original/current
CoreCLR Mono six and JCW one all passed, with current raw/project agreement.
K_NTLM_COMPILER_ADMITTED=yes bash "$SESSION_FILES/build-final-ntlm-coreclr.sh"andK_NTLM_QUIET_ADMITTED=yes bash "$SESSION_FILES/run-final-ntlm-coreclr.sh":passed raw original/current three cases using the existing
noexclusions=trueopt-in and on-device loopback only. Same-property projectcommand correctly returned explicit empty-selection infrastructure rejection
(native exit 8, command exit 1), with no fresh project TRX. This is not positive
NTLM project-mode coverage or a claim that prior device files were absent.
K_JCW_AOT_QUIET_ADMITTED=yes bash "$SESSION_FILES/run-final-jcw-nativeaot.sh":fresh original/current actual NativeAOT both blocked by SIGABRT after
JavaAbstractMethodTest starts, without completed TRX/counts. Not a passing
47-case gate. Current matching debug ELF resolves RhFailFast; the dump does
not independently prove exact
ic.NewMethod. Original final debug ELF wasnot retained, so no original named-frame claim is substituted.
Exact identity/category/multiplicity, native UID/wire/TRX and independent project
outcomes reconcile. Current registries contain 79/34/5 fixtures with 52/24/2
incomplete descriptors: bounded reflection remains. Actual migrated APK
declarations and Mono ILC inputs are NUnit/TestRunner.Core-free; Mono/JCW
raw-to-project package changes are exactly 13 TRX satellites, not unchanged APKs.
Normal builds retain documented warnings, including user-selected original
GetType/MSTEST0036, boxed AreNotSame and unchanged TimeZone assertions. Unknown
earlier GCBridge event failure, compiler assertions, root regression, alias error
and unsuccessful attempts remain preserved. No exclusions, weakened assertions,
GC overrides or JCW runtime fix were used to turn failures into passes.
No JCW AOT support, selected-category AOT coverage, additional ABI/runtime, or
fresh Azure/device-CI certification is claimed.