Skip to content

Add pre-PR consistency checks to the preset submission workflow #4746

Description

@mnriem

Maintainer-only. This issue concerns the repository-owned preset submission agentic workflow and its compiled lock file.

Problem

.github/workflows/add-community-preset.md validates submission fields, documentation, and a release download before requesting a catalog PR. Some consistency checks are specified only as agent instructions, while others do not compare the published preset manifest, submission fields, README, and generated catalog entry with one another. The workflow also applies validation-passed before it has checked the generated catalog and documentation changes.

Requested changes

  1. Check the published artifact. After the existing pinned-URL and archive-download checks, read the relevant preset.yml from that downloaded archive. Compare its preset ID, version, Spec Kit requirement, and required extensions with the issue fields and the catalog entry being prepared. Check that the release tag agrees with the submitted and archived versions. Support presets located in a monorepo subdirectory. Do not execute downloaded content.
  2. Check README install references. Preserve the currently accepted install-command forms, including --dev. If the submitted preset's README also contains a release-install --from URL for that preset, check it against the submitted Download URL; another accepted command must not mask a contradictory URL.
  3. Check generated files before requesting a PR. Verify that the catalog JSON parses, its entry reflects the validated values, preset IDs remain alphabetically ordered, and the documentation row has the expected values and name order. For updates, verify that created_at is preserved.
  4. Change when success is recorded. Apply validation-passed and request the draft PR only after these checks succeed. Fix an agent-generated catalog or documentation error and recheck it before proceeding; do not label it a submitter defect. Report a confirmed submission mismatch on the issue without requesting a PR. Continue to distinguish an unavailable tool or archive fetch as blocked, rather than treating it as a mismatch or a pass.
  5. Make the comparisons reproducible. Implement mechanically verifiable checks in a tested helper or the repository's existing workflow-validation mechanism rather than relying only on additional prose instructions. Regenerate the workflow lock file from the source workflow.

Acceptance criteria

  • A matching issue, archive manifest, README, and generated entry can still produce a draft catalog PR.
  • A version or required-extension mismatch between the issue and published manifest does not produce a PR.
  • A stale --from URL for the submitted preset is reported even if the README also contains an accepted --dev command; a README containing only a valid --dev command retains its current behavior.
  • Invalid ordering or metadata in generated catalog/documentation files is corrected and rechecked before PR creation.
  • A blocked archive fetch produces neither validation-passed nor a PR and does not attribute the environmental failure to the submitter.

Out of scope: Changing maintainer-applied label triage, handling manually opened catalog PRs, or reviewing third-party preset code.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triage-nice-to-haveVerdict: evidence-backed fix or greenlit feature — land after review

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions