Skip to content

Make service updates durable and rollback-safe - #165

Merged
comfuture merged 3 commits into
mainfrom
fix/163-durable-service-updates
Sep 4, 2026
Merged

comfuture merged 3 commits into
mainfrom
fix/163-durable-service-updates

Conversation

@comfuture

@comfuture comfuture commented Sep 4, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • install exact @codori/server bundles under each registered service metadata directory with bounded npm execution, --no-audit, --no-fund, and no global package mutation
  • replace the registered service launcher with a stable service-owned Node bootstrap whose atomically replaced active-bundle.json selects an absolute validated entrypoint
  • run updates through an exact-version worker that stages before restart, verifies loopback HTTP and the exact running/durable version, and restores the previous known-good bundle on failure
  • launch Linux workers as transient systemd-run units so restarting the main service cannot kill the update transaction with its cgroup
  • expire abandoned update leases; reconcile a serving target as healthy, record interrupted downloads as failed, and let the bootstrap atomically restore the previous bundle after an abandoned restart
  • persist durable update phases (downloading, restarting, healthy, failed, and rolled-back), active version, failure reason, and structured update.log diagnostics
  • remove startup-time registry adoption and nested npx, while preserving systemd, launchd, and Windows scope/home/host/port/Tailscale launch behavior
  • stop the browser completion watcher on durable failure/rollback and expose the persisted reason in the update tooltip

Architecture rationale

The service manager continues to point at its existing run-service.sh or run-service.cmd, but that launcher now invokes a small service-owned bootstrap. The bootstrap reads one atomically replaced selection file and launches the selected bundle absolute entrypoint with the recorded Node runtime. The atomic selection file is the single launch source of truth, while service.json records current/previous bundle and status for inspection.

Legacy registrations are migrated without taking the listener down. The worker first prepares a rollback copy of the currently running exact package, writes a current-version selection and stable launcher, then switches the selection to the already validated target and restarts. A preparation failure leaves the registered launcher and running process unchanged.

On Linux, the update worker runs in a separate transient systemd unit rather than the main service cgroup. A five-minute durable lease also prevents interrupted downloading or restarting states from locking updates forever. If a restart was abandoned, the bootstrap restores previous-bundle.json atomically before launching.

Commits

  • 2ccf0b4 — fix: make service updates rollback-safe
  • c7fc853 — test: cover durable service update transactions
  • 2d50a0b — fix: supervise durable service update workers

Validation

  • pnpm --filter @codori/server test — 261 passed; Linux smoke skipped on macOS
  • pnpm lint — passed
  • pnpm typecheck — passed
  • pnpm test — client 609, server 261, WebXR 191 passed locally
  • pnpm build — passed for client, WebXR, server, and CLI
  • pnpm --filter @codori/server pack --pack-destination <temp> — passed; tarball contains service-bundle.js, service-update.js, and service-update-worker.js
  • git diff --check — passed
  • GitHub Actions CI run 33867922878 passed on exact head 2d50a0b, including the Linux real-process smoke

The Linux smoke launches a real temporary listener twice through the generated launcher, checks its process tree and exact version endpoint, and uses no npx path. No live m5 or chunsik service was modified or restarted.

Risks and safeguards

  • Initial registration and legacy migration require the exact published package to be stageable. Failure occurs before service-manager mutation, preserving the existing launcher and listener.
  • Health verification is bounded and requires both the running package and durable selection to match the target. A wrong version, exit, bind failure, or timeout triggers a previous-bundle restart and verification.
  • Cleanup retains the active and previous bundles; it runs only after a confirmed healthy target or confirmed rollback.
  • Cross-platform service-manager behavior is unchanged; launcher rendering and lifecycle coverage exercise macOS, Linux, and Windows paths.

Closes #163

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-04T11:10:31.395734Z c7fc853 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c7fc853325

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/server/src/service-update.ts
Comment thread packages/server/src/service-update.ts
@comfuture
comfuture merged commit e59167d into main Sep 4, 2026
2 checks passed
@comfuture
comfuture deleted the fix/163-durable-service-updates branch September 4, 2026 12:35
@comfuture comfuture mentioned this pull request Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make service updates durable and rollback-safe

1 participant