Shell API and CLI: Add option for staging additional runtime targets by nathanwilliams-ct · Pull Request #2147 · apache/buildstream

nathanwilliams-ct · 2026-07-06T13:31:13Z

This enables users to add additional functionality such as debug tooling in the shell sandbox,
without needing to modify the target element. This is achived through introducing a new option to
shell.

All existing API and UX is maintained, to not break existing scripts. An alternative design was
considered, to have the additional elements as positional arguments similar to the existing element,
but this would need manual parsing to handle the cases where -- is present and not
present, splitting based on a .bst suffix. This UX could be re-visited in future.

Example usage:
bst shell --with base.bst example.bst -- cat example.txt
Where:

example.bst is a simple import element with no dependencies that imports a file called example.txt
base.bst provides a basic alpine sysroot with a standard set of unix tooling (sh, df, cat etc).

Changes:

Introduces test_with_other_targets integration test to the shell test suite.
Adds --with cli option to the shell subcommand and updates it's documentation.
- option can be used multiple times by caller, providing a list of targets.
Extends the shell top level calling interface in Buildstream core to accept a list of
other_targets
- This is where the targets are loaded into elements and checked to make sure they are present
Extends the shell element implementation to accept a list of other targets
- This is where the other elements are staged and integrated into the sandbox

towards: #422

nathanwilliams-ct · 2026-07-10T14:36:34Z

Thanks, whoever triggered the CI... looks like it needs a little work, despite the tests working locally in the tox environment and I missed the linters.

nathanwilliams-ct · 2026-07-16T12:42:57Z

I think this should pass CI now, fixed docs, linters, static type checker, formatter and 3.10 issues

nathanwilliams-ct · 2026-07-16T14:50:02Z

pre-commit hooks would be nice. I keep running the commands locally, but It seems I am doing them in the wrong order.. fixing one thing breaks the others XD

nathanwilliams-ct · 2026-07-16T14:54:15Z

tox
tox -- --integration
tox -e docs
tox -e lint
tox -e mypy
tox -e format

And the tests locally only run on 313 and 314 for me, while CI does also 310 311 312

Is complete list right?

juergbi · 2026-07-17T07:58:04Z

And the tests locally only run on 313 and 314 for me, while CI does also 310 311 312

tox runs the tests on all Python versions from 3.10 to 3.14 by default, but skipping versions where the interpreter is not installed. It's normally fine to test against just one Python version locally, letting CI take care of the others, unless you're actively debugging a version-specific issue.

Is complete list right?

CI also runs buildgrid and buildbarn, tests against master of buildstream-plugins-community, and builds wheels. Unless you're working in these specific areas, it shouldn't be necessary to run those locally.

nathanwilliams-ct · 2026-07-17T08:30:57Z

Ah there's a subtle conflict between format and the docs generation.

nathanwilliams-ct · 2026-07-17T09:15:45Z

Static type checking moved out into it's own PR: #2153

juergbi · 2026-09-11T11:41:37Z

+
+            target_loader._set_fullpath_overrides(target_name, temp_target_file.name)
+
+            self.build([target])


This should no longer be necessary with the fullpath override, right?

Changing the dependencies messes with the cache key so a build is necessary still.

Direct runtime dependencies shouldn't affect the cache key. And for build shells (where we add build dependencies), we shouldn't invoke a build before entering the shell anyway. Or am I misremembering or missing something? Have you confirmed a cache key issue with the latest version of your branch (with the fullpath override)?

hmmm, No, I will do some further testing.

juergbi · 2026-09-11T11:49:25Z

Nit: We prefer limiting normal text lines in git commit messages to 72 characters, matching usual git conventions.

juergbi · 2026-09-11T13:45:38Z

The test seems to be failing on Python 3.10 (Ubuntu 22.04) and Python 3.11 (Debian 12).

This enables users to add additional functionality such as debug tooling in the shell sandbox, without needing to modify the target element. This is achived through introducing a new option to shell. All existing API and UX is maintained, to not break existing scripts. An alternative design was considered, to have the additional elements as positional arguments similar to the existing element, but this would need manual parsing to handle the cases where `--` is present and not present, splitting based on a `.bst` suffix. This UX could be re-visited in future. Example usage: bst shell --with base.bst example.bst -- cat example.txt Where: - example.bst is a simple import element with no dependencies that imports a file called example.txt - base.bst provides a basic alpine sysroot with a standard set of unix tooling (sh, df, cat etc). Changes: - Introduces `test_with_other_targets` integration test to the shell test suite. - Adds `--with` cli option to the shell subcommand and updates it's documentation. - option can be used multiple times by caller, providing a list of targets. - Extends the shell top level calling interface in Buildstream core to accept a list of other_targets - This is where the targets are loaded into elements and checked to make sure they are present - Extends the shell element implementation to accept a list of other targets - This is where the other elements are staged and integrated into the sandbox

juergbi

Thanks for the updates. I think it's almost ready but have a few minor comments.

juergbi · 2026-10-09T13:22:25Z

+import sys
+import tempfile


src/buildstream/_loader/loader.py:18:0: W0611: Unused import sys (unused-import) src/buildstream/_loader/loader.py:19:0: W0611: Unused import tempfile (unused-import)

juergbi · 2026-10-09T13:24:44Z

-                ignore_project_artifact_remotes=ignore_project_artifact_remotes,
-                ignore_project_source_remotes=ignore_project_source_remotes,
-            )
+            if other_targets:


Either here or in Stream.shell_with() we should reject the combination of --use-buildtree and --with with a clear error message as this combination can't work, right?

juergbi · 2026-10-09T13:31:33Z

+    def _set_fullpath_override(self, filename: str, fullpath: str | None):
+        if fullpath:
+            self._fullpath_overrides[filename] = fullpath
+        else:
+            self._fullpath_overrides.pop(filename, None)


Nit: Each branch is a one-liner and used exactly once. Wouldn't it be simpler to inline it at the call site?

If you think it's clearer to have this in a separate method (it could hypothetically also be used for other purposes), then I think this should be a context manager to provide some actual benefit at the call site.

Either variant is fine with me.

juergbi · 2026-10-09T13:34:07Z

+# Test staging and running additional targets in the shell of the main target for debugging.
+@pytest.mark.datafiles(DATA_DIR)
+@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+def test_with_other_targets(cli, datafiles):


I'd like to see at least a minimal test for a build shell as well.

juergbi · 2026-10-09T13:34:38Z

+            else:
+                raise StreamError(
+                    "Only BUILD and RUN scopes are supported",
+                    detail="Use the --build and --use-buildtree options to shell into a build tree",


This detail doesn't make sense to me. Leftover from copy paste?

nathanwilliams-ct requested review from BenjaminSchubert, abderrahim, cs-shadow, gtristan and juergbi as code owners July 6, 2026 13:31

nathanwilliams-ct marked this pull request as draft July 10, 2026 14:35

nathanwilliams-ct force-pushed the nathan/shell-multi branch 4 times, most recently from 6c82853 to fb98811 Compare July 16, 2026 10:47

nathanwilliams-ct marked this pull request as ready for review July 16, 2026 12:42

nathanwilliams-ct force-pushed the nathan/shell-multi branch 3 times, most recently from 5c266ec to 4857d9e Compare July 16, 2026 14:48

nathanwilliams-ct force-pushed the nathan/shell-multi branch 2 times, most recently from abffc99 to 5a52dae Compare July 17, 2026 09:15

juergbi reviewed Jul 17, 2026

View reviewed changes

Comment thread src/buildstream/element.py Outdated

nathanwilliams-ct force-pushed the nathan/shell-multi branch from 5a52dae to 9e9eba9 Compare August 3, 2026 12:02

nathanwilliams-ct force-pushed the nathan/shell-multi branch from 9e9eba9 to 311b613 Compare August 18, 2026 13:52

nathanwilliams-ct force-pushed the nathan/shell-multi branch from 311b613 to e271424 Compare September 1, 2026 13:14

nathanwilliams-ct force-pushed the nathan/shell-multi branch from e271424 to c843357 Compare September 8, 2026 13:58

nathanwilliams-ct commented Sep 9, 2026

View reviewed changes

Comment thread src/buildstream/_stream.py Outdated

nathanwilliams-ct commented Sep 9, 2026

View reviewed changes

Comment thread tests/integration/shell.py

nathanwilliams-ct commented Sep 9, 2026

View reviewed changes

Comment thread src/buildstream/_loader/loader.py Outdated

nathanwilliams-ct force-pushed the nathan/shell-multi branch from 1a38816 to 031710f Compare September 9, 2026 15:30

juergbi reviewed Sep 11, 2026

View reviewed changes

Comment thread src/buildstream/_loader/loader.py Outdated

juergbi reviewed Sep 11, 2026

View reviewed changes

Comment thread src/buildstream/_frontend/cli.py

juergbi reviewed Sep 11, 2026

View reviewed changes

Comment thread src/buildstream/_stream.py Outdated

nathanwilliams-ct force-pushed the nathan/shell-multi branch from af385fc to 502eee7 Compare September 15, 2026 13:42

juergbi reviewed Sep 18, 2026

View reviewed changes

Comment thread src/buildstream/_loader/loader.py Outdated

Add app type hint to cli shell function

7a4607f

nathanwilliams-ct force-pushed the nathan/shell-multi branch from c79e587 to c5a7569 Compare September 29, 2026 09:41

nathanwilliams-ct force-pushed the nathan/shell-multi branch from c5a7569 to b275ed5 Compare October 7, 2026 10:17

nathanwilliams-ct force-pushed the nathan/shell-multi branch from b275ed5 to a3cd988 Compare October 7, 2026 10:28

juergbi reviewed Oct 9, 2026

View reviewed changes


		target_loader._set_fullpath_overrides(target_name, temp_target_file.name)

		self.build([target])

Conversation

nathanwilliams-ct commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nathanwilliams-ct commented Jul 10, 2026

Uh oh!

nathanwilliams-ct commented Jul 16, 2026

Uh oh!

nathanwilliams-ct commented Jul 16, 2026

Uh oh!

nathanwilliams-ct commented Jul 16, 2026

Uh oh!

juergbi commented Jul 17, 2026

Uh oh!

nathanwilliams-ct commented Jul 17, 2026

Uh oh!

nathanwilliams-ct commented Jul 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

juergbi commented Sep 11, 2026

Uh oh!

juergbi commented Sep 11, 2026

Uh oh!

Uh oh!

Uh oh!

juergbi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nathanwilliams-ct commented Jul 6, 2026 •

edited

Loading