Smoke tests failing in stage 2 with `VSTESTTASK2` error

[mthalman]

VSTESTTASK2 : error Message:  [/vmr/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.csproj]

No other information exists. It's an error contained in the build output but not an actual test case failure. Example build (internal link)

It would seem this is introduced by new functionality from within vstest that gets compiled into the SDK and used in the stage 2 build.

[mthalman]

This may be a regression caused by microsoft/vstest#4834. These tests were working last Friday and failing today after incorporating changes that included microsoft/vstest#4834. @nohwnd - Can you take a look?
Here's a binlog: BuildTests_0122193259.zip. We need to get this resolved soon for .NET 9 Preview 1.

[mthalman]

I ran a test build (internal link) that reverted the change from microsoft/vstest#4834 but that didn't help at all.

[nohwnd]

This is more likely linked to this change: https://github.com/dotnet/sdk/pull/38098/files which changes how failed tests are reported to take advantage of msbuild terminal logger which is now the default experience for dotnet build.

I am trying to repro to see if the smoke tests failed and VSTEST fails to report the details of the failure. Or if VSTEST is failing and the change needs to be reverted.

[nohwnd]

Running another test build with opt-out from the new feature https://dev.azure.com/dnceng/internal/_build/results?buildId=2360131&view=results

In the meantime I am trying to repro locally.

[nohwnd]

I can't repro fully.

I can run the tests but I always get the full output of the error. And I can't run tests without failures locally because I don't have all the artifacts. I always see: Tarball path '' specified in does not exist. ( from here ) I downloaded what I could but I still cannot get it to work locally.

I can see in binary logger the same error output as in build, but it is followed by the actual error message.

In your run I cannot see any failures in trx, so I am not sure where the errors are coming from and if they are real.

I've edited the target to actually propagate the opt-out flag and started another build here (from a new branch, I want to see how the other one will finish) https://dev.azure.com/dnceng/internal/_build/results?buildId=2360247&view=results

[MichaelSimons]

[Triage] closing as is appears this fix flowed in.