release/13.6: site build time doubled after the full 13.6 API snapshot was added; internal build hits its 60-minute timeout

[eerhardt]

Problem

On deploy-vnext-release, which picks up merges from release/13.6, the Run Aspire Publish step of the internal Aspire.Dev-Build pipeline (definition 1564) went from about 27 minutes to 57.5 minutes.

Internal build 3087220 ran out the job's 60-minute timeout. It was cancelled during 🔒 Publish to Pipeline Artifact, after aspire publish itself had succeeded.

The ~20 builds before it (on both deploy and deploy-vnext-release) all finished this step in 24–29 minutes.

Likely cause

Build 3087220 is the first one to include 44b9819bb ("docs: ingest the complete pinned Aspire 13.6 API snapshot"). That commit added about 1.34M lines of API reference JSON under src/frontend/src/data/pkgs/ and a 227k-line src/frontend/src/data/twoslash/aspire.d.ts. The number of generated pages more than doubled.

A related commit, 201a9af63 ("ci: give the complete API build an 8 GiB Node heap"), only updated .github/workflows/frontend-build.yml. The internal Azure DevOps pipeline didn't get the bigger heap. It didn't run out of memory this time, but it has little margin.

Timing breakdown

This compares build 3086870 (previous, commit 003b9367) with build 3087220 (commit 3b1bbcfb), taken from the aspire publish --pipeline-log-level debug output:

Phase	3086870	3087220	Change
Pages built	15,311	32,142	2.1×
Vite bundling	2m 27s	5m 47s	2.4×
Astro page build (generating static routes → [build] Complete!)	10m 17s	17m 16s	1.7×
[build] Complete! → StaticHost -> .../publish/	15m 34s	37m 36s	2.4×
Container image build + tarball	1m 13s	2m 22s	1.9×
Total aspire publish	27m 20s	57m 29s	2.1×

Largest time sink: after Astro finishes, inside dotnet publish of StaticHost

The single biggest block is a 37.6-minute stretch with no log output, between Astro printing [build] Complete! and MSBuild printing StaticHost -> .../publish/.

This time is spent inside dotnet publish src/statichost/StaticHost/StaticHost.csproj, after the frontend dist/ exists. It is most likely ASP.NET static web assets processing over tens of thousands of files:

• discovering every file under dist/
• fingerprinting each file
• building the asset and endpoint manifests
• Brotli-compressing each file (PublishCompressionFormats=brotli)
• Caching.targets (SetAstroImmutableCacheHeaders), which runs UpdateStaticWebAssetEndpoints and item Remove/Include over the full StaticWebAssetEndpoint list

This phase grew faster than the page count (2.4× vs 2.1×), which suggests the cost rises faster than linearly with the number of files.

Other signals

• At the end of the step the agent reported Free disk space on / is lower than 5%; Currently used: 95.45%. Larger dist/ and Brotli output plus a bigger image tarball may be slowing disk I/O.
• The Pagefind index grew from 15.3k to 32.3k HTML files (about 42s to 64s). That's a small share of the total.

Suggested next steps

• Unblock: raise timeoutInMinutes on the "Build Projects and Ev2 ARM templates" job so release builds finish.
• Diagnose: capture an MSBuild binlog (-bl) of the StaticHost publish to find which target or task uses the ~37 minutes.
• Reduce static web assets cost: for example, skip fingerprinting/compression for HTML or API reference pages, compress during the Astro build instead, rework Caching.targets so it doesn't rescan the full endpoint list, or exclude large generated trees from static web asset discovery.
• Reduce page count: check whether every API reference page (for example, the full provisioning API surface) needs to be pre-rendered, or whether some could be combined or loaded on demand.
• Apply the 8 GiB Node heap from 201a9af63 to the internal pipeline as well.
• Look into disk space on the macOS agent (clean up intermediate outputs, or use a bigger agent).