Skip to content

GH-1283: Optimize number of build compilations in CI - #1284

Open
xborder wants to merge 15 commits into
apache:mainfrom
xborder:ci-build-once-test-many
Open

xborder wants to merge 15 commits into
apache:mainfrom
xborder:ci-build-once-test-many

Conversation

@xborder

@xborder xborder commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

What's Changed

  • Build Java artifacts once with JDK 17 and reuse them across the test matrix.
  • Build C Data JNI artifacts once and reuse them across JDK 17, 21, and 25.
  • Distribute compiled classes and Maven artifacts through GitHub Actions artifacts.
  • Run prebuilt tests directly with Surefire to avoid recompilation.
  • Preserve Vector allocator and memory-core custom test executions.
  • Add checks to ensure test classes are restored and no unexpected compilation occurs.
  • Keep pure-Java tests running on Ubuntu, macOS Intel, macOS ARM, and Windows.
  • Reduce duplicate compilation while retaining the existing platform and JDK test coverage.
  • Ran a few tests, managed to reduce the total runner time by ~50%

Closes #1283.

This PR was assisted by AI

@github-actions

This comment has been minimized.

@xborder
xborder marked this pull request as ready for review September 5, 2026 04:45
@kou kou added the chore PRs that make misc changes. label Sep 6, 2026
@github-actions github-actions Bot added this to the 20.0.0 milestone Sep 6, 2026
@jbonofre

jbonofre commented Sep 6, 2026

Copy link
Copy Markdown
Member

Thanks! I gonna take a look.

@xborder

xborder commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

@jbonofre can you also take a look at this one?

@xborder

xborder commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

@kou since you took a look at my last 2 CI changes, can you also take a look at this one?

Comment thread .github/workflows/test.yml Outdated
path: java-build.tgz
retention-days: 1
- name: Exclude reactor artifacts from Maven dependency cache
run: rm -rf .docker/maven-cache/repository/org/apache/arrow

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to remove them? We can't use them to avoid rebuilding Arrow Java, right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is removed so the cache doesn’t carry old Arrow binaries into future runs. The binaries are already in the build artifacts used by every test job

Comment thread .github/workflows/test.yml Outdated
Comment on lines -96 to -92
- arch: AArch64
jdk: 17
macos: latest

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that we need os: macos-XXX here.

@xborder xborder Oct 1, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure I understood. Isn't that what was done here?

Comment thread .github/workflows/test.yml
Comment on lines +115 to +121
- name: Cache Docker Volumes
if: ${{ matrix.compose_service }}
uses: actions/cache@v6
with:
path: .docker/maven-cache
key: java-build-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: java-build-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why doe we need this?
download-artifact isn't enough?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this avoids downloading external dependencies again in each job. download-artifacts only contains the compiled Arrow output and installed org/apache/arrow artifacts

Comment thread .github/workflows/test.yml Outdated
@xborder
xborder force-pushed the ci-build-once-test-many branch from 18a8b8a to 5309167 Compare October 1, 2026 17:45
@xborder
xborder requested a review from kou October 1, 2026 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore PRs that make misc changes.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI] Build Java and C Data artifacts once

3 participants