MSBuild crashing in the build

[jkoritzinsky]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=410579
Build error leg or test failing: Build / Mono Product Build windows x64 release / Build product
Pull request: #91768

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "Child Node \"[0-9]+\" exited prematurely.",
  "BuildRetry": true,
  "ExcludeConsoleLog": false
}

Report

Build	Repository	Step Name	Console log	Pull Request
1515969	dotnet/runtime	Build product	Log	#130907
1512190	dotnet/runtime	Build product	Log	#128799
1512091	dotnet/runtime	Build product	Log	#130755
1511605	dotnet/runtime	Copy native test components to test output folder	Log	#130813
1510658	dotnet/runtime	Build Tests	Log	#130642
1507086	dotnet/runtime	Restore and Build Product	Log
1505679	dotnet/runtime	Build	Log	#130356
3020456	dotnet-runtime	Build product	Log
1501559	dotnet/runtime	Build product	Log
3017238	dotnet-runtime	Build product	Log
1495674	dotnet/runtime	Build product	Log	#130259
3015375	dotnet-runtime	Build product	Log
1483740	dotnet/runtime	Build Tests	Log

Build	Repository	Test	Pull Request
1508598	dotnet/runtime	WBT-NoWebcil-MONO-ST-Wasm.Build.Tests.RebuildTests.WorkItemExecution	#130701
1508282	dotnet/runtime	WBT-NoWebcil-MONO-ST-Wasm.Build.NativeRebuild.Tests.SimpleSourceChangeRebuildTest.WorkItemExecution	#130478
1508136	dotnet/runtime	WBT-NoWorkload-CLR-ST-Wasm.Build.Tests.RebuildTests.WorkItemExecution	#129634
1506826	dotnet/runtime	WBT-Workloads-MONO-ST-Wasm.Build.Tests.DebugLevelTests.WorkItemExecution	#130478
1506689	dotnet/runtime	WBT-Workloads-MONO-ST-Wasm.Build.Tests.RebuildTests.WorkItemExecution	#130589
1505604	dotnet/runtime	Wasm.Build.Tests.WasmNativeDefaultsTests.DefaultsWithBuild
1502840	dotnet/runtime	Wasm.Build.Tests.WasmNativeDefaultsTests.WasmNativeStripDefaultWithPublish	#130122
1504785	dotnet/runtime	Wasm.Build.NativeRebuild.Tests.FlagsChangeRebuildTests.ExtraEmccFlagsSetButNoRealChange
1501868	dotnet/runtime	WBT-NoWebcil-MONO-ST-Wasm.Build.Tests.SatelliteAssembliesTests.WorkItemExecution	#130420
1500817	dotnet/runtime	WBT-Workloads-MONO-ST-Wasm.Build.Tests.WasmNativeDefaultsTests.WorkItemExecution	#128304
1500083	dotnet/runtime	WBT-NoWebcil-MONO-ST-Wasm.Build.Tests.FilesToIncludeInFileSystemTests.WorkItemExecution	#130104
1500001	dotnet/runtime	WBT-Workloads-MONO-ST-Wasm.Build.Tests.RebuildTests.WorkItemExecution	#130363
1497814	dotnet/runtime	WBT-NoWorkload-CLR-ST-Wasm.Build.Tests.RebuildTests.WorkItemExecution	#130097
1492365	dotnet/runtime	WBT-Workloads-MONO-ST-Wasm.Build.Tests.InvariantGlobalizationTests.WorkItemExecution	#130051
1487375	dotnet/runtime	WBT-NoWebcil-MONO-ST-Wasm.Build.Tests.MainWithArgsTests.WorkItemExecution
1480147	dotnet/runtime	WBT-NoWorkload-CLR-ST-Wasm.Build.Tests.LazyLoadingTests.WorkItemExecution
1479819	dotnet/runtime	WBT-Workloads-MONO-ST-Wasm.Build.Tests.PreloadingTests.WorkItemExecution	#129543
1477535	dotnet/runtime	WBT-NoWebcil-MONO-ST-Wasm.Build.Tests.WasmNativeDefaultsTests.WorkItemExecution	#129748
1476579	dotnet/runtime	Wasm.Build.Tests.DllImportTests.CallIntoLibrariesWithNonAlphanumericCharactersInTheirNames(config: Release, aot: False, libraryNames: ["with-hyphen", "with#hash-and-hyphen", "with.per.iod", "with🚀unicode#"])	#129727
1475084	dotnet/runtime	WBT-Workloads-MONO-ST-Wasm.Build.Tests.InvariantGlobalizationTests.WorkItemExecution	#129232

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	6	33

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=410579
Error message validated: Child Node "[0-9]+" exited prematurely.
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 9/19/2023 6:06:29 PM UTC

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=410579
Build error leg or test failing: Build / Mono Product Build windows x64 release / Build product
Pull request: #91768

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "Child Node \"[0-9]+\" exited prematurely.",
  "BuildRetry": true,
  "ExcludeConsoleLog": false
}

Author:	jkoritzinsky
Assignees:	-
Labels:	blocking-clean-ci, area-Infrastructure, untriaged, Known Build Error
Milestone:	-

[ViktorHofer]

190 hits per month

cc @rainersigwald @rokonec

[lewing]

reup for 97 hits per month

[MichalStrehovsky]

We now have more useful information in the logs for these. Two of the available logs have this in it:

2024-02-19T21:13:25.8115210Z MSBUILD : error MSB4166: Child node "2" exited prematurely. Shutting down. Diagnostic information may be found in files in "/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/MSBuildTemprunner/" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
2024-02-19T21:13:25.8126770Z MSBUILD : error MSB4166: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/MSBuildTemprunner/MSBuild_pid-50918_4ee39488d5ca4c7d9fb049f19f43c144.failure.txt:
2024-02-19T21:13:25.8128200Z MSBUILD : error MSB4166: UNHANDLED EXCEPTIONS FROM PROCESS 50918:
2024-02-19T21:13:25.8129250Z MSBUILD : error MSB4166: =====================
2024-02-19T21:13:25.8140670Z MSBUILD : error MSB4166: 2/19/2024 9:13:25 PM
2024-02-19T21:13:25.8241810Z MSBUILD : error MSB4166: System.IO.IOException: No buffer space available
2024-02-19T21:13:25.8260730Z MSBUILD : error MSB4166:  ---> System.Net.Sockets.SocketException (55): No buffer space available
2024-02-19T21:13:25.8363710Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.WriteCore(ReadOnlySpan`1 buffer)
2024-02-19T21:13:25.8364490Z MSBUILD : error MSB4166:    --- End of inner exception stack trace ---
2024-02-19T21:13:25.8465500Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.WriteCore(ReadOnlySpan`1 buffer)
2024-02-19T21:13:25.8569320Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.Write(Byte[] buffer, Int32 offset, Int32 count)
2024-02-19T21:13:25.8670730Z MSBUILD : error MSB4166:    at Microsoft.Build.BackEnd.NodeEndpointOutOfProcBase.RunReadLoop(Stream localReadPipe, Stream localWritePipe, ConcurrentQueue`1 localPacketQueue, AutoResetEvent localPacketAvailable, AutoResetEvent localTerminatePacketPump)
2024-02-19T21:13:25.8773480Z MSBUILD : error MSB4166: ===================

@rainersigwald does this ring any bell?

[rainersigwald]

All of the recent hits seem to be dotnet/msbuild#10342, which should be fixed in the preview7 GA build. Is there a timeline for updating to that? Anything my team can help with?