Output device messages are invisible in Visual Studio at the default logging level (server mode)

[timcassell]

Describe the bug

In server mode, ServerModePerCallOutputDevice.DisplayAsync maps TextOutputDeviceData, FormattedTextOutputDeviceData and SessionMessageOutputDeviceData to LogLevel.Information, and warnings/errors to Warning/Error. Visual Studio's Output → Tests pane renders only the Warning and Error messages at its default Logging level = Informational; the informational text appears only once the level is raised to Diagnostic.

So output a framework writes through the documented user-facing channel is invisible to the user by default, where the same content was visible when the framework ran under VSTest.

This surfaced while reviewing a Microsoft.Testing.Platform adapter for BenchmarkDotNet — dotnet/BenchmarkDotNet#3229 — whose entire benchmark report is written at run level and therefore disappeared from Test Explorer's output, having been visible through the project's existing VSTest adapter.

Version used

Microsoft.Testing.Platform 2.4.1 (the latest at the time of writing); also reproduced on 2.3.3. Visual Studio 18.10.1, .NET 10 test host, Windows 10 22H2.

Steps To Reproduce

1. Write a Microsoft.Testing.Platform test framework that calls IOutputDevice.DisplayAsync(this, new TextOutputDeviceData("hello from the run"), cancellationToken) while executing a request.
2. In Visual Studio, set Tools → Options → All Settings → Test → Logging level to Informational (Default).
3. Run the tests from Test Explorer and open Output → Tests.
4. Set the logging level to Diagnostic and run again.

A concrete repro is the adapter in dotnet/BenchmarkDotNet#3229: running any benchmark from Test Explorer writes BenchmarkDotNet's full report through IOutputDevice.

Expected behavior

Messages written to the output device are shown to the user by default, consistent with #4213, which moved them onto the client precisely because they are user-facing:

the logs are meant to be troubleshooting oriented … On the other hand, the output devices messages are designed as user facing messaging and are currently not forwarded to be printed into the client (e.g. Test Explorer).

Actual behavior

They arrive as client/log at Information, which the IDE shows only in its troubleshooting view. Captured off the wire on 2.4.1, one run of the adapter above sends 139 client/log notifications: 127 Information, 6 Warning, 2 Error, 4 Trace. At the default logging level Visual Studio shows exactly the 8 warnings and errors; the summary table, the build progress and the per-benchmark measurements are all among the 127 informational ones that are dropped.

At Diagnostic the same run shows everything (Visual Studio 18.10.1, MTP 2.4.1):

[12:16:12.288 PM]  [MyTests.dll] | Method   | Value   | Mean     | Error |
[12:16:12.289 PM]  [MyTests.dll] | Identity | async-1 | 42.30 æs |    NA |
[12:16:12.289 PM]  | Method   | Value   | Mean     | Error |
[12:16:12.290 PM]  |--------- |-------- |---------:|------:|
[12:16:12.290 PM]  | Identity | async-1 | 42.30 μs |    NA |

Additional context

This makes the VSTest bridge lossy in one direction. ObjectModelConverters preserves per-test output faithfully — TestResultMessage.StandardOutCategory becomes StandardOutputProperty, attachments become FileArtifactProperty, and VS renders both. But MessageLoggerAdapter.SendMessage has nowhere equivalent to put run-level messages:

case TestMessageLevel.Informational:
    _logger.LogInformation(message);
    _outputDevice.DisplayAsync(this, new TextOutputDeviceData(message), _cancellationToken).Await();
    break;

Under VSTest the _messageLogger?.SendMessage(...) above it still reaches the Tests pane at the default level; under MTP the same call becomes an informational client/log the user does not see. Any framework reporting at run level rather than per test loses that output in VS on migration.

Possible resolutions, in your hands:

• forward output-device text at a level clients surface by default, or add a distinct user-output notification separate from client/log;
• if Information is correct and the gap is in Visual Studio's mapping of it, saying so here would be enough — the report then belongs on Developer Community, and a statement to cite would help;
• failing either, document in the protocol docs what a client is expected to render, so framework authors know run-level output needs another channel (per-test StandardOutputProperty, or file artifacts).

Secondary observation. At Diagnostic every line appears twice, once prefixed with the test host's file name and once bare, because ProxyOutputDevice writes to both the terminal device and the server-mode device and Visual Studio captures the child's stdout as well as the client/log. The stdout copy is also mis-decoded, visible in the excerpt above: 42.30 æs from stdout against 42.30 μs from client/log, and likewise 1 æs : 1 Microsecond against 1 μs : 1 Microsecond in the legend. The duplication may be by design; the encoding looks worth a look on whichever side owns it.

_{🤖 Investigated with Claude Code.}

[MihuBot]

I'm a bot. Here is a possible related and/or duplicate issue (I may be wrong):

• How to output message on VS TestExplorer window #5999

[Evangelink]

Thanks for the detailed investigation. I want to clarify the distinction we need to preserve here.

IOutputDevice output and diagnostic logs are intentionally different channels:

• Output-device messages are user-facing. They are commonly localized and should be visible under the client's normal/default presentation settings. They can still carry informational, warning, or error semantics.
• Logs are diagnostic. They are intended for framework/platform/extension troubleshooting, generally are not localized, and should remain controlled by diagnostic logging verbosity.

That separation was the intent behind #4213 and #4178: stop forwarding the platform's diagnostic logger to Test Explorer and forward output-device messages instead. The remaining problem is that the server-mode JSON-RPC protocol currently serializes those output-device messages through the log-shaped client/log notification. Its payload contains only a log level and message, so the client cannot know that an Information message came from the user-facing output device rather than from diagnostics. Visual Studio consequently applies its logging-level filtering and hides it at the default level.

We should not solve this by promoting ordinary output to Warning, nor by making all informational diagnostics visible by default. Both would erase the distinction above in a different way.

A proper fix will require a server-mode protocol change that preserves the user-output category end to end—for example, a distinct display/output notification or an explicit discriminator, negotiated through protocol version/capability support. There is already a similar DisplayMessage concept in the separate dotnet test pipe protocol, but no equivalent currently exists in the JSON-RPC server-mode protocol.

We do want to fix this. It will require coordination with the vstest/Visual Studio client side, compatibility behavior for clients that do not yet support the new message, and some time for the change to flow through the different supported Visual Studio versions. We will use this issue to track that work and coordination.

The duplicate output and encoding observation may be a separate issue; it should not drive us toward conflating user output with diagnostic logging.