Skip to content

[probe, do not merge] What may GITHUB_TOKEN do to reviews on a protected base - #148

Closed
FeliLucero1 wants to merge 1 commit into
mainfrom
felipe/probe-review-dismissal
Closed

FeliLucero1 wants to merge 1 commit into
mainfrom
felipe/probe-review-dismissal

Conversation

@FeliLucero1

Copy link
Copy Markdown
Contributor

Throwaway probe for the verdict_mode decision in #147. Not for merge — this branch and PR get deleted as soon as the run is read.

Measures, with the real GITHUB_TOKEN on a PR targeting protected main:

  1. can github-actions[bot] submit REQUEST_CHANGES? (baseline does this today, so: control)
  2. can it dismiss a review it authored itself? — the narrower alternative to judge mode
  3. can it submit APPROVE at all? — what judge mode in [CXH-2780] Let CI submit an approval, not only a block (verdict_mode) #147 depends on

🤖 Generated with Claude Code

Temporary, not for merge. Measures three things the auto-dismiss vs judge
decision turns on: whether the bot can dismiss a review it authored, whether
branch protection blocks that, and whether the bot can submit APPROVE at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probe: request changes

@github-actions
github-actions Bot dismissed their stale review October 11, 2026 18:53

probe: can the author of a review dismiss it?

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probe: approve

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Superseded — see the current review report for commit 7cf7ad6ba73a

General PR Review: [probe, do not merge] What may GITHUB_TOKEN do to reviews on a protected base

Blocking Issues: 0 | Suggestions: 2 | Threads Resolved: 0
Criteria: Criteria status: none loaded - .claude/skills/ci-review.md was not found at trusted base ce5fc28e14cb.
Review mode: full
View review run

Review Summary

This PR adds one temporary workflow, .github/workflows/probe-review-dismissal.yaml. It runs on same-repo PRs whose head branch starts with felipe/probe-review-dismissal. Using GITHUB_TOKEN, it tries three things on the PR's reviews, which matches the stated intent:

  • submit a REQUEST_CHANGES review
  • dismiss that same review
  • submit an APPROVE review

It then logs every review's state.

I scanned the full diff for security and correctness. Permissions are kept to contents: read / pull-requests: write. PR number, repo and SHA reach the shell only through env:, and head_ref is used only in the job if: expression, so there is no script injection. No repo-local criteria loaded, so only the base criteria were applied. Nothing in the diff was left unreviewed.

Security Issues

None found.

Correctness Issues

None found.

Suggestions

  • New (medium confidence) .github/workflows/probe-review-dismissal.yaml:58-60: if probe 3 works, the bot's APPROVE stays on the PR and nothing dismisses it. Where Actions are allowed to approve PRs, that approval can count toward required reviews on protected main until the PR is closed.
  • New (low confidence) .github/workflows/probe-review-dismissal.yaml:62-63: RESULT 4 lists every review on the PR. That includes reviews general-pr-review posts as github-actions[bot], and extra probe reviews from synchronize re-runs, which makes the result easy to misread.
Prompt for AI agents
Verify each finding against the current code and only fix it if needed.

## Suggestions

In `.github/workflows/probe-review-dismissal.yaml`:
- Around line 58-60: Capture the review id returned by the APPROVE probe, for example `AID=$(printf '%s' "${LAST_OUT}" | jq -r '.id // empty')`. If it is non-empty, PUT `repos/${REPO}/pulls/${PR}/reviews/${AID}/dismissals` with a message and log the rc. That way a successful bot approval does not stay on the protected-base PR, and the run also shows whether the bot can dismiss an APPROVE.
- Around line 62-63: Filter the final review listing down to reviews this run created. Either select by the ids from steps 1 and 3, or by `.body | startswith("probe:")`. Then reviews from the general-pr-review workflow and from earlier `synchronize` runs cannot confuse RESULT 4.

Comment on lines +58 to +60
probe "3. submit APPROVE" POST "repos/${REPO}/pulls/${PR}/reviews" \
-f commit_id="${SHA}" -f event=APPROVE -f body="probe: approve"
echo "RESULT 3 submit APPROVE: rc=${LAST_RC}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion (medium confidence): if probe 3 works, github-actions[bot]'s APPROVE stays on the PR, and nothing here dismisses it. If the repo allows Actions to approve PRs, that approval can count toward required reviews on protected main until the PR is closed. Consider dismissing the APPROVE review id right after logging RESULT 3 (this also tells you whether the bot can dismiss an APPROVE as well as a CHANGES_REQUESTED).

Comment on lines +62 to +63
probe "4. final review states" GET "repos/${REPO}/pulls/${PR}/reviews"
echo "RESULT 4 states=$(printf '%s' "${LAST_OUT}" | jq -r '[.[]|{id,state,user:.user.login}]|tostring' 2>/dev/null)"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion (low confidence): general-pr-review also runs on this PR and submits reviews as github-actions[bot]. Probe 4's list can therefore mix in that workflow's REQUEST_CHANGES/COMMENT reviews, and a synchronize re-run adds more probe reviews on top. Filter by the review ids this run created (or by body prefix probe:) so RESULT 4 can't be misread.

@FeliLucero1

Copy link
Copy Markdown
Contributor Author

Probe done, results captured in #147. Conclusive on a PR targeting protected main, with plain GITHUB_TOKEN and no extra config:

result
bot submits REQUEST_CHANGES works (control)
bot dismisses a review it authored works — review went to DISMISSED
bot submits APPROVE works — review went to APPROVED

The docs' protected-branch caveat ("you must be a repository administrator or be included in the list of people or teams who can dismiss") did not bite: that restriction only applies when a dismissal allow-list is actually configured, and none of our repos configure one.

Closing without merging; the workflow file existed only on this branch.

@FeliLucero1
FeliLucero1 deleted the felipe/probe-review-dismissal branch October 11, 2026 18:54
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

General PR Review: [probe, do not merge] What may GITHUB_TOKEN do to reviews on a protected base

Blocking Issues: 0 | Suggestions: 2 | Threads Resolved: 0
Criteria: Criteria status: none loaded - .claude/skills/ci-review.md was not found at trusted base ce5fc28e14cb.
Review mode: full
View review run

Review Summary

This PR adds one temporary workflow, .github/workflows/probe-review-dismissal.yaml. It runs on same-repo PRs whose head branch starts with felipe/probe-review-dismissal. Using GITHUB_TOKEN, it tries three things on the PR's reviews, which matches the stated intent:

  • submit a REQUEST_CHANGES review
  • dismiss that same review
  • submit an APPROVE review

It then logs every review's state.

I scanned the full diff for security and correctness. Permissions are kept to contents: read / pull-requests: write. PR number, repo and SHA reach the shell only through env:, and head_ref is used only in the job if: expression, so there is no script injection. No repo-local criteria loaded, so only the base criteria were applied. Nothing in the diff was left unreviewed.

Security Issues

None found.

Correctness Issues

None found.

Suggestions

  • New (medium confidence) .github/workflows/probe-review-dismissal.yaml:58-60: if probe 3 works, the bot's APPROVE stays on the PR and nothing dismisses it. Where Actions are allowed to approve PRs, that approval can count toward required reviews on protected main until the PR is closed.
  • New (low confidence) .github/workflows/probe-review-dismissal.yaml:62-63: RESULT 4 lists every review on the PR. That includes reviews general-pr-review posts as github-actions[bot], and extra probe reviews from synchronize re-runs, which makes the result easy to misread.
Prompt for AI agents
Verify each finding against the current code and only fix it if needed.

## Suggestions

In `.github/workflows/probe-review-dismissal.yaml`:
- Around line 58-60: Capture the review id returned by the APPROVE probe, for example `AID=$(printf '%s' "${LAST_OUT}" | jq -r '.id // empty')`. If it is non-empty, PUT `repos/${REPO}/pulls/${PR}/reviews/${AID}/dismissals` with a message and log the rc. That way a successful bot approval does not stay on the protected-base PR, and the run also shows whether the bot can dismiss an APPROVE.
- Around line 62-63: Filter the final review listing down to reviews this run created. Either select by the ids from steps 1 and 3, or by `.body | startswith("probe:")`. Then reviews from the general-pr-review workflow and from earlier `synchronize` runs cannot confuse RESULT 4.

Reviewed commit: 7cf7ad6ba73a

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found — see the full review report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant