Run MSTest acceptance assets under both reflection and source generation#9370
Draft
Evangelink wants to merge 1 commit into
Draft
Run MSTest acceptance assets under both reflection and source generation#9370Evangelink wants to merge 1 commit into
Evangelink wants to merge 1 commit into
Conversation
Add shared infrastructure so each generated acceptance-test asset is built twice (reflection + MSTest.SourceGeneration) and the same assertions run against both metadata paths: - MetadataMode enum and variant-aware TestHost.LocateFrom (bin vs bin/SourceGen). - AcceptanceSourceGen helper injects the generator via CustomBeforeMicrosoftCommonProps into an isolated bin/SourceGen output, resolves the packed version, and exposes a global kill-switch. The generator reference is conditioned off net4x because source generation is .NET-only. - TestAssetFixtureBase builds the second (source-gen) variant by default, with a SkipSourceGenVariant opt-out. - (tfm, MetadataMode) DynamicData sources on AcceptanceTestBase; source-gen only pairs with .NET TFMs. Pilots converted and validated (build.cmd -pack): InconclusiveTests (single-TFM) and TestRunParametersTests (multi-TFM incl. net462) — 26/26 pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the acceptance-test infrastructure so each generated MSTest acceptance asset can be exercised under two metadata mechanisms: the default runtime reflection path and an opt-in MSTest.SourceGeneration-driven path, with test matrices parameterized to run the same behavioral assertions against both modes.
Changes:
- Introduces
MetadataModeand sharedAcceptanceSourceGenhelpers to build an isolated “SourceGen” variant of generated test assets. - Updates test-asset build + test-host location logic to be metadata-mode-aware (
bin/...vsbin/SourceGen/...). - Converts two pilot acceptance suites to run across
(tfm, metadataMode)matrices.
Show a summary per file
| File | Description |
|---|---|
| test/Utilities/Microsoft.Testing.TestInfrastructure/TestHost.cs | Adds metadataMode parameter and updates host discovery to use bin/SourceGen for source-gen builds. |
| test/Utilities/Microsoft.Testing.TestInfrastructure/TestAssetFixtureBase.cs | Adds default-on second build variant with source generation injected; exposes variant build status/diagnostics. |
| test/Utilities/Microsoft.Testing.TestInfrastructure/MetadataMode.cs | Adds enum representing the metadata mechanism (reflection vs source generation). |
| test/Utilities/Microsoft.Testing.TestInfrastructure/AcceptanceSourceGen.cs | Adds MSBuild props injection + output isolation logic for building source-gen variants of assets. |
| test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestRunParametersTests.cs | Switches to (tfm, metadataMode) DynamicData and locates host based on mode. |
| test/IntegrationTests/MSTest.Acceptance.IntegrationTests/InconclusiveTests.cs | Cross-products lifecycle steps with metadata modes and runs assertions against both paths. |
| test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/Helpers/AcceptanceTestBase.cs | Adds shared DynamicData sources for (tfm, metadataMode) matrices and a global kill-switch integration. |
Copilot's findings
- Files reviewed: 7/7 changed files
- Comments generated: 2
Comment on lines
+63
to
+72
| if (!SkipSourceGenVariant && !AcceptanceSourceGen.IsGloballyDisabled && result.ExitCode == 0) | ||
| { | ||
| string sourceGenArgs = await AcceptanceSourceGen.PrepareBuildArgumentsAsync(testAsset.TargetAssetPath); | ||
| DotnetMuxerResult sourceGenResult = await DotnetCli.RunAsync( | ||
| $"build {testAsset.TargetAssetPath} -c Release {sourceGenArgs}", | ||
| callerMemberName: $"{assetName}_SourceGen", | ||
| cancellationToken: cancellationToken); | ||
| SourceGenBuildResult = sourceGenResult; | ||
| SourceGenVariantBuilt = sourceGenResult.ExitCode == 0; | ||
| } |
Comment on lines
+131
to
135
| string expectedRootPath = metadataMode == MetadataMode.SourceGeneration | ||
| ? Path.Combine(rootFolder, "bin", AcceptanceSourceGen.OutputSubFolder, buildConfiguration.ToString(), tfm) | ||
| : Path.Combine(rootFolder, "bin", buildConfiguration.ToString(), tfm); | ||
| string[] executables = Directory.GetFiles(expectedRootPath, moduleName, SearchOption.AllDirectories); | ||
| string? expectedPath = executables.SingleOrDefault(p => p.Contains(rid) && p.Contains(verb == Verb.publish ? "publish" : string.Empty)); |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| // (reflection and, unless globally disabled, source generation) so the same expectations | ||
| // validate both metadata paths. | ||
| public static IEnumerable<object[]> LifecycleAndMetadataModes { get; } | ||
| = (from lifecycle in Enum.GetValues<Lifecycle>() |
| // validate both metadata paths. | ||
| public static IEnumerable<object[]> LifecycleAndMetadataModes { get; } | ||
| = (from lifecycle in Enum.GetValues<Lifecycle>() | ||
| from mode in MetadataModesToRun |
| /// (net4x) are paired with <see cref="MetadataMode.Reflection"/> only. | ||
| /// </summary> | ||
| public static IEnumerable<object[]> AllTfmsAndMetadataModes { get; } | ||
| = (from tfm in TargetFrameworks.All |
| /// </summary> | ||
| public static IEnumerable<object[]> AllTfmsAndMetadataModes { get; } | ||
| = (from tfm in TargetFrameworks.All | ||
| from mode in MetadataModesToRun |
| /// <see cref="MetadataModesToRun"/> mode. | ||
| /// </summary> | ||
| public static IEnumerable<object[]> NetTfmsAndMetadataModes { get; } | ||
| = (from tfm in TargetFrameworks.Net |
| /// </summary> | ||
| public static IEnumerable<object[]> NetTfmsAndMetadataModes { get; } | ||
| = (from tfm in TargetFrameworks.Net | ||
| from mode in MetadataModesToRun |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds shared test infrastructure so each generated MSTest acceptance-test asset is built twice — once normally (runtime reflection) and once with the
MSTest.SourceGeneration(AOT reflection-metadata) generator enabled — and the same behavioral assertions run against both metadata paths. This extends the broad acceptance coverage to the source-generatedReflectionMetadataHookpath, which today is only exercised by a handful of dedicated tests.This is the foundation + two validated pilots; the mechanical roll-out to the remaining acceptance classes will follow in batches.
How (central shared-infra, minimal per-file churn)
MetadataMode { Reflection, SourceGeneration }enum.AcceptanceSourceGenhelper — injects the generator via an MSBuildCustomBeforeMicrosoftCommonPropsfile passed only on the source-gen build, into an isolatedbin/SourceGen+obj/SourceGenoutput (so both hosts coexist). Resolves the packedMSTest.SourceGenerationversion and exposes a global kill-switch env var (MSTEST_ACCEPTANCE_SKIP_SOURCEGEN).TestAssetFixtureBasebuilds the second variant by default, with aSkipSourceGenVariantopt-out for assets source generation can't support.TestHost.LocateFromis variant-aware (binvsbin/SourceGen).(tfm, MetadataMode)DynamicDatasources onAcceptanceTestBase.Source generation is .NET-only
The generated metadata references
ModuleInitializerAttribute/DynamicallyAccessedMembersAttribute/DynamicDependencyAttribute, which don't compile on .NET Framework. So:PackageReferenceis conditioned!$(TargetFramework.StartsWith('net4')), so a multi-targeting asset's net4x leg builds as plain reflection (and isn't exercised in source-gen mode).Validation
After
build.cmd -pack, both converted pilots pass — 26/26:InconclusiveTests— single-TFM (NetCurrent),Lifecycle × MetadataMode(16 cases).TestRunParametersTests— multi-TFM incl. net462 (10 cases): net462 reflection-only, net8.0/net10.0 both modes.Follow-up
Mechanically thread
MetadataModethrough the remaining ~80 acceptance classes in batches, addingSkipSourceGenVariant/opt-outs as source-gen gaps surface (inherited[TestClass], generics, cross-assembly reflection, VSTest-host/NativeAOT/Aspire/Playwright/ServerMode assets).Co-authored-by: Copilot