Make service updates durable and rollback-safe #163

Summary

Make a Codori service update durably switch the registered launcher to the new server bundle before restarting, and keep a bounded rollback path if the new bundle cannot become healthy.

On a real Linux 0.15.0-to-0.15.1 update, the update command restarted the service but left its launcher as unversioned npx --yes @codori/server start. Because an older global codori-server binary was still on PATH, the service started 0.15.0 again. Startup adoption then launched a nested npx @codori/server@0.15.1 install before binding HTTP. The systemd unit remained active while port 4310 was closed and the public endpoint returned 502 for roughly eight minutes.

The nested install eventually started a transient 0.15.1 child, but the global 0.15.0 installation and service launcher remained unchanged. The next restart can repeat the same network-dependent adoption and outage.

Background

Issue #128 fixed the browser-side completion watcher after a service update, but deliberately left server-side download and registration behavior out of scope. This issue addresses that server-side handoff.

The failure was captured on a user-level systemd service with Node 22.22.2 and npm 10.9.7. The exact host name and private URL are omitted; the relevant behavior is reproducible from the generated launcher, process tree, npm log, listener state, and version endpoint.

Current implementation facts

• packages/server/src/service-update.ts#createUpdateCommand runs npx --yes @codori/server@latest service restart ... after a one-second delay.
• service restart restarts the existing registered unit; it does not install the new package into a durable service-owned location or pin the launcher to the selected version.
• packages/server/src/service.ts generates launchers that always execute unversioned npx --yes @codori/server start ....
• On the affected host, that unversioned launch resolved the still-installed global @codori/server 0.15.0 binary.
• packages/server/src/cli.ts#execAdoptedPackage detects 0.15.1 and spawns another npx --yes @codori/server@0.15.1 start ... before the HTTP listener binds.
• checkStartupUpdate() can fall back only after execPackage rejects. There is no timeout or readiness boundary for a child that stays alive while npm installs or stalls.
• The live process tree was systemd -> npx unversioned -> 0.15.0 codori-server -> npx @codori/server@0.15.1 -> 0.15.1 codori-server.
• During the nested install, systemd reported the unit as active while nothing listened on 4310. The npm debug log paused in dependency installation/audit and recorded a 503 response from the registry audit endpoint.
• After the listener appeared, GET /api/service/update reported installed 0.15.1, but the durable global package was still 0.15.0 and run-service.sh was unchanged.
• Existing startup-adoption tests mock execPackage as resolving or rejecting. They do not exercise a hanging child, the generated launcher's actual resolution, HTTP readiness, durable version selection, or rollback.

Problem statement

The current update flow restarts first and downloads/chooses the target version on the critical startup path. A service manager can therefore report success while Codori is unavailable. A transient adopted child can also make the version endpoint look updated without changing what the next service restart will execute.

This violates both availability and update durability: success does not mean the registered service now owns a known target bundle.

Goal

Stage and verify the target @codori/server bundle before stopping the current listener, atomically make the registered service launch that prepared version, and confirm health after restart. If preparation or startup fails, keep or restore the last known-good bundle and expose a durable failure reason.

Normal service starts must not require an npm registry request or nested npx adoption before binding HTTP.

Scope

• Replace unversioned network-resolved service launch with a durable, service-owned exact bundle/entrypoint.
• Stage package download/install while the old server is still serving, with npm audit/fund disabled for this runtime installation path.
• Validate the staged manifest, Node compatibility, executable entrypoint, and exact requested version before changing the launcher.
• Atomically update launcher/metadata to the staged version, restart, and perform a bounded local health/version check.
• Roll back the launcher and restart the last known-good version when the target does not become healthy before the deadline.
• Make update status distinguish downloading, restarting, healthy completion, failed/rolled back, and the durable active version.
• Remove or strictly bound startup adoption so it cannot wait indefinitely or mask a stale durable launcher.
• Add focused unit/integration tests and a real Linux service smoke test.

Non-goals

• Changing the Codex app-server project inventory behavior; track that separately.
• Requiring a global npm install -g or mutating the user's unrelated global packages.
• Treating systemd/launchd process existence as HTTP readiness.
• Reporting success merely because an exact-version child process was spawned.
• Deleting the last known-good bundle during the same update transaction.

Suggested direction

1. Give each registered service a managed bundle directory under its existing metadata directory. Install the target exact version into a staging directory with the recorded Node/npm toolchain and --ignore-scripts only if compatible with the package's required runtime behavior; always use --no-audit --no-fund and a bounded timeout.
2. Resolve and validate the staged @codori/server package manifest plus codori-server entrypoint. Record the exact version and absolute entrypoint in versioned metadata.
3. Generate the launcher against that absolute, service-owned entrypoint rather than npx @codori/server. Replace launcher/metadata atomically only after staging succeeds.
4. Restart the unit and poll loopback /api/service/update (or a dedicated health endpoint) until it reports the exact target version. Keep the current browser completion watcher from Reload the dashboard after a service update finishes #128, but make it consume the richer durable state.
5. If readiness times out or the process exits, restore the previous launcher/metadata and restart the previous bundle. Append structured diagnostics to update.log.
6. Retain at least the current and previous known-good bundles. Garbage-collect older versions only after a later confirmed healthy start.
7. If startup adoption remains as a compatibility bridge for old launchers, give it a short timeout and fail open to the installed server. It must not be the normal steady-state launcher after one successful update.

Acceptance criteria

• Requesting an update downloads and validates the exact target bundle while the current service remains reachable.
• A failed, slow, or unavailable registry/audit endpoint cannot take down the currently running version.
• The registered launcher and metadata identify an exact service-owned bundle after a successful update; they do not execute unversioned npx @codori/server.
• The service does not perform an npm registry lookup or nested package installation on an ordinary restart.
• Update success is reported only after loopback HTTP is healthy and the running server reports the exact durable target version.
• A target that hangs, exits, binds the wrong port, or reports the wrong version is bounded by a timeout and rolls back to the prior healthy bundle.
• After a successful 0.15.0-to-0.15.1 update, stopping and starting the unit offline launches 0.15.1 directly with no 0.15.0 parent or nested npx process.
• systemd, launchd, and Windows service launcher generation retain their existing scope, home, host, port, and Tailscale behavior.
• Unit tests cover staging failure, hanging preparation, atomic selection, target readiness, wrong-version response, rollback, and cleanup.
• A Linux smoke test verifies the process tree, listener continuity/downtime bound, version endpoint, launcher contents, and offline second restart.
• Server lint, typecheck, tests, and production packaging pass.

Suggested smoke test

1. Register a user-level service on version N while a different older global codori-server is on PATH.
2. Publish or point the test registry at N+1, trigger the dashboard update, and delay/fail the package audit endpoint.
3. Verify version N remains reachable throughout staging and no systemd-active/port-closed window occurs before the prepared restart.
4. Allow staging to complete, verify the launcher switches to the exact managed N+1 entrypoint, and confirm the loopback version/health response before UI reload.
5. Stop network access, restart the service twice, and verify N+1 starts directly both times.
6. Repeat with an N+1 entrypoint that hangs or exits; verify bounded rollback to N and actionable update.log output.

References

• Follow-up to Reload the dashboard after a service update finishes #128, whose server-side update mechanism was a non-goal.
• The stale launcher was originally generated by the service workflow introduced around Rebuild cross-platform service registration with a codori service command, switchable project root, and update prompts #110.
• The independent project-catalog and initial discovery behavior is tracked in Restore reliable Codex App project discovery #162.