The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2904404	dotnet-runtime	Send job to Helix (Windows)	Log
1294857	dotnet/machinelearning	Run Helix Tests	Log
2904264	dotnet-performance	Send job to Helix (Windows)	Log
2904241	dotnet-runtime	Generate CORE_ROOT	Log
2904148	dotnet-dotnet	Build (centOSStreamContainer)	Log
2904157	dotnet-runtime	Send job to Helix (Windows)	Log
1294546	dotnet/runtime	Send tests to Helix (Unix)	Log
2903801	dotnet-maui	Controls (v18.4) Image,ImageButton,IndicatorView,InputTransparent,IsEnabled,IsVisible	Log
1294498	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#65428
1294410	dotnet/runtime	Send tests to Helix (Windows)	Log
1294440	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#123515
1294424	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#120114
1294458	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124396
1294485	dotnet/sdk	Run AoT Tests in Helix	Log	dotnet/sdk#53038
2904014	dotnet-efcore	Send job to helix	Log
1294446	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124365
1294382	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123909
2903984	dotnet-runtime	Send job to Helix (Unix)	Log
1294352	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#124202
1294360	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123588
1294366	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124362
2904001	dotnet-efcore	Send job to helix	Log
1294357	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#124408
1294411	dotnet/roslyn	Test	Log	dotnet/roslyn#82408
1294376	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#124173
2903999	dotnet-aspire	🟣Restore (linux_arm64)	Log
1294349	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124411
1294318	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124405
1294373	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124397
1294309	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#124410
1294392	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#53031
1294303	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123744
1294365	dotnet/maui	Run Integration Tests - Build windows	Log	dotnet/maui#34051
1294279	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124142
1294133	dotnet/runtime	Send tests to Helix (Unix)	Log
1294306	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124409
1294372	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#53033
1294256	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124307
1294065	dotnet/maui	Controls Layout	Log
1294237	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117256
1294326	dotnet/maui	Run Helix Tests	Log	dotnet/maui#34050
1294232	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124303
2903946	dotnet-performance	Send job to Helix (Unix)	Log
1294187	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123921
1294265	dotnet/sdk	Run FullFramework Tests	Log	dotnet/sdk#53036
1294230	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#52960
1294082	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#124117
1294094	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124400
1294167	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124395
1294059	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119432
1294174	dotnet/performance	Send job to Helix (Windows)	Log	dotnet/performance#5119
2903833	dotnet-roslyn	Test	Log
1294033	dotnet/runtime	Send tests to Helix (Unix)	Log
2903783	dotnet-maui	DeviceTestsMacCatalyst (Windows)	Log
1294063	dotnet/runtime	Send to Helix	Log	dotnet/runtime#108158
1294113	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#52641
1294035	dotnet/runtime	Send tests to Helix (Unix)	Log
1293989	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#124361
1293995	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#124394
2903782	dotnet-maui	Controls (v18.4) Entry	Log
1294079	dotnet/maui	Controls Layout	Log
1294105	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#53033
1293831	dotnet/runtime	Send to Helix	Log	dotnet/runtime#111514
1294053	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#65426
1294054	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#65426
1293955	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124397
1294080	dotnet/maui	Run Helix Tests	Log	dotnet/maui#29755
1293964	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124399
1294019	dotnet/aspnetcore	Run Quarantined Tests	Log	dotnet/aspnetcore#65415
2903785	dotnet-roslyn	Test	Log
1294023	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#52959
1293943	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124347
2903724	dotnet-runtime	Send job to Helix (Windows)	Log
1293779	dotnet/runtime	Publish	Log	dotnet/runtime#123145
2903570	dotnet-maui	Controls (v18.4) ListView	Log
2903740	dotnet-efcore	Send job to helix	Log
1293851	dotnet/runtime	Send tests to Helix (Windows)	Log
1293846	dotnet/runtime	Send to Helix (Windows)	Log	dotnet/runtime#124391
1293934	dotnet/performance	Send job to Helix (Windows)	Log	dotnet/performance#5093
2903447	dotnet-runtime	Send to Helix	Log	#57891
1293805	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116193
2903702	dotnet-performance	Send job to Helix (Windows)	Log
2903705	dotnet-efcore	Send job to helix	Log
1293877	dotnet/performance	Send job to Helix (Unix)	Log
1293843	dotnet/machinelearning	Run Helix Tests	Log	dotnet/machinelearning#7580
1293857	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#52960
1293750	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#124387
1293691	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123932
1293769	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124390
1293703	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#124173
1293795	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#65427
1293818	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#53031
1293796	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#65427
1293645	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122396
1293742	dotnet/maui	Controls ScrollView	Log
1293654	dotnet/runtime	Send to Helix (Windows)	Log	dotnet/runtime#124258
1293748	dotnet/aspnetcore	Run Quarantined Tests	Log	dotnet/aspnetcore#65415
2903639	dotnet-performance	Send job to Helix (Windows)	Log
1293661	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124200
1293613	dotnet/runtime	Send to Helix	Log	dotnet/runtime#124388
Displaying 100 of 2895 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
17	613	2895

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez