Skip to content

Add hourly production smoke check - #270

Closed
admdly wants to merge 1 commit into
mainfrom
ops/guard-deploy-migrations
Closed

admdly wants to merge 1 commit into
mainfrom
ops/guard-deploy-migrations

Conversation

@admdly

@admdly admdly commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

The catalogue outage was worker code deployed without its D1 migration (0026), so every public list read returned 500 DATABASE_ERROR.

Since production deploys run through Workers Builds — which does not apply D1 migrations — no repo-side step can gate deploys. This hits the exact failing read path hourly (GET /extensions/v2/extensions?limit=1, asserts a result array) so a recurrence is caught quickly. No secrets needed; manual dispatch supported.

Deliberately not included: chaining migrations into cf-deploy (never executes under Workers Builds) and a PR-time pending-migration gate (pending is the expected state on migration PRs). The deploy-time migration step belongs in the Workers Builds build config.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
🔒 Security Review ✅ Completed 2026-10-07T01:28:54.659882Z 1eed1c4 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
api ea28de7 Oct 07 2026, 01:30 AM

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/workflows/monitor.yml">

<violation number="1" location=".github/workflows/monitor.yml:21">
P2: The smoke check has no time bounds: curl runs without `--max-time`/`--connect-timeout` and the job has no `timeout-minutes`. A hung or stalled connection keeps the job running up to GitHub's 360-minute default and hourly runs overlap, so a recurrence can go undetected exactly when the check is most needed. Add `--max-time 30 --connect-timeout 10` to curl and `timeout-minutes: 10` to the job.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

steps:
- name: Public extension list returns rows
run: |
body=$(curl -sS --fail-with-body "https://api.fossbilling.net/extensions/v2/extensions?limit=1" -H "Accept: application/json")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The smoke check has no time bounds: curl runs without --max-time/--connect-timeout and the job has no timeout-minutes. A hung or stalled connection keeps the job running up to GitHub's 360-minute default and hourly runs overlap, so a recurrence can go undetected exactly when the check is most needed. Add --max-time 30 --connect-timeout 10 to curl and timeout-minutes: 10 to the job.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/monitor.yml, line 21:

<comment>The smoke check has no time bounds: curl runs without `--max-time`/`--connect-timeout` and the job has no `timeout-minutes`. A hung or stalled connection keeps the job running up to GitHub's 360-minute default and hourly runs overlap, so a recurrence can go undetected exactly when the check is most needed. Add `--max-time 30 --connect-timeout 10` to curl and `timeout-minutes: 10` to the job.</comment>

<file context>
@@ -0,0 +1,22 @@
+    steps:
+      - name: Public extension list returns rows
+        run: |
+          body=$(curl -sS --fail-with-body "https://api.fossbilling.net/extensions/v2/extensions?limit=1" -H "Accept: application/json")
+          echo "$body" | jq -e '.result | type == "array"' > /dev/null
</file context>
Suggested change
body=$(curl -sS --fail-with-body "https://api.fossbilling.net/extensions/v2/extensions?limit=1" -H "Accept: application/json")
body=$(curl -sS --fail-with-body --max-time 30 --connect-timeout 10 "https://api.fossbilling.net/extensions/v2/extensions?limit=1" -H "Accept: application/json")

The catalogue outage was worker code deployed without its D1 migration. Workers Builds does not apply migrations, so no repo-side step can gate deploys — this hits the exact public read path hourly so a recurrence is caught quickly.
@admdly
admdly force-pushed the ops/guard-deploy-migrations branch from 1eed1c4 to ea28de7 Compare October 7, 2026 01:29
@admdly admdly changed the title Add hourly production smoke check and document deploy migration order Add hourly production smoke check Oct 7, 2026
@admdly

admdly commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Superseded: we already have HetrixTools externally. A repo-side cron adds nothing without its own alerting, and a status-code check on / would not even have caught this outage (the error page returned 200). Instead, adding a content-aware HetrixTools monitor (see thread).

@admdly admdly closed this Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant