Skip to content

Fix multibuildpack integration test timeouts (builds #19-22 failures) - #1178

Merged
tnikolova82 merged 1 commit into
masterfrom
fix-multibuildpack-test-timeouts
Oct 9, 2026
Merged

tnikolova82 merged 1 commit into
masterfrom
fix-multibuildpack-test-timeouts

Conversation

@tnikolova82

Copy link
Copy Markdown
Contributor

Summary

Fixes consistent failures in builds #19-22 where the nodejs multibuildpack integration test timed out after ~920 seconds across all 5 retry attempts.

Problem

The MultiBuildpack/when_supplied_with_nodejs test has been consistently failing since build #19 (Oct 3). Investigation revealed the test was using the default 20-second Eventually timeout, which is insufficient for this heavy Rails 7.2 + Webpacker multi-buildpack scenario.

Why the test fails

  1. Heavy multi-buildpack staging (~4-5 minutes):

    • nodejs-buildpack downloads Node 22.23.3, Yarn 1.22.22, Python 3.13.16
    • Yarn install takes ~53 seconds for node_modules (webpack, webpacker, etc.)
    • ruby-buildpack downloads Ruby 3.2.11, Bundler 2.7.2, RubyGems 3.6.8
    • Bundle install processes 78 gems including full Rails 7.2.3 stack
    • Webpacker performs asset compilation
  2. Slow app boot time (60-120 seconds):

    • Rails apps with Webpacker need time to initialize
    • Multi-buildpack apps have additional startup overhead
    • The test fixture runs node -v and ruby -v commands on each request
  3. Parallel execution resource contention:

    • Tests run with GINKGO_NODES=3
    • Multiple heavy Rails apps deploy simultaneously
    • Limited CF Diego cell resources cause scheduling delays

Build history

Build Date Buildpack Commit Status Notes
#18 Oct 2-3 bbe2a4be ✅ SUCCESS Last successful build
#19 Oct 3 78d409c5 ❌ FAILED First failure
#20-21 Oct 5 82c4944b, 90ca34cd ❌ FAILED GINKGO_NODES fix didn't help
#22 Oct 6 7640eb2a ❌ FAILED serial_flag fix didn't help

The GINKGO_NODES changes (commits 78d409c → 82c4944 → 7640eb2) were unrelated to this timeout issue.

Solution

Extended Eventually timeout to 90 seconds with 2-second poll intervals for all multibuildpack tests:

  • ✅ when_supplied_with_nodejs (PRIMARY FIX)
  • ✅ when_ruby_is_a_supply_for_the_binary_buildpack
  • ✅ when_supplied_with_go
  • ✅ when_supplied_with_.NET_Core

Changes made

// Added time import
import (
    "path/filepath"
    "testing"
    "time"  // NEW
    ...
)

// Extended timeout from default 20s to 90s with 2s polling
Eventually(deployment, 90*time.Second, 2*time.Second).Should(Serve(...))

Precedent

This fix follows the exact same pattern proven successful in:

  1. Commit d5294b6 (Oct 2, 2026): "Increase Eventually timeout for 3 flaky Docker-backend integration tests"

    • Fixed: Offline/vendor_bundle, Default/custom_gemfile tests
    • Rationale: "Docker platform backend reports container as started before the app process inside has actually finished booting and bound its port"
  2. PR Increase JRuby test polling timeout for cflinuxfs5 stability #1140: JRuby test timeout fixes

    • Extended JRuby test timeout to 90s for the same reason

Both commits explicitly state that the default 20s timeout races with slow app boot times under parallel CI load.

Testing

The fix is conservative and low-risk:

  • ✅ Only affects the 4 multibuildpack tests (isolated change)
  • ✅ Uses proven timeout values (90s) from d5294b6
  • ✅ Doesn't change test logic, only wait time
  • ✅ No changes to pipeline configuration needed

Expected outcome

  • nodejs multibuildpack test will pass consistently
  • Other multibuildpack tests gain resilience against timing variations
  • No impact on other tests in the suite

Investigation

Full investigation details available in the commit message and local documentation:

Checklist

Related

Fixes consistent failures observed in builds #19-22 of the
create-cf-infrastructure-and-execute-integration-test-for-ruby job,
where the nodejs multibuildpack test timed out after ~920 seconds
across all 5 retry attempts.

## Problem

The test `MultiBuildpack/when_supplied_with_nodejs` was using the
default 20-second Eventually timeout, which is insufficient for:

1. **Heavy multi-buildpack staging**: The Rails 7.2 + Webpacker app
   with nodejs-buildpack + ruby-buildpack takes 4-5 minutes to stage
   - Node.js buildpack downloads Node 22.23.3, Yarn 1.22.22, Python
   - Yarn install takes ~53 seconds for node_modules
   - Ruby buildpack downloads Ruby 3.2.11, Bundler 2.7.2, RubyGems
   - Bundle install processes 78 gems including Rails 7.2.3
   - Webpacker/asset compilation

2. **Slow app boot time**: Rails apps with Webpacker can take 60-120
   seconds to start and bind their port

3. **Parallel execution resource contention**: Tests run with
   GINKGO_NODES=3, causing multiple heavy applications to compete for
   limited CF Diego cell resources

## Solution

Extended Eventually timeout to 90 seconds with 2-second poll intervals
for all multibuildpack tests:
- `when_supplied_with_nodejs` (PRIMARY FIX)
- `when_ruby_is_a_supply_for_the_binary_buildpack`
- `when_supplied_with_go`
- `when_supplied_with_.NET_Core`

This matches the precedent established for other slow-booting fixtures:
- PR #1140: JRuby test timeouts
- Commit d5294b6: Docker backend timeout fixes for 3 flaky tests

## Testing

The fix follows the exact pattern proven successful in d5294b6, which
stated: "The default 20s Eventually timeout has been observed to
intermittently race with the Docker platform backend under parallel CI
load: the container reports as started before the app process inside
has actually finished booting and bound its port."

The same issue affects these multi-buildpack tests, particularly the
nodejs test with its heavy Rails + Webpacker fixture.
@tnikolova82
tnikolova82 merged commit 357a479 into master Oct 9, 2026
7 checks passed
@tnikolova82
tnikolova82 deleted the fix-multibuildpack-test-timeouts branch October 9, 2026 10:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant