Skip to content

Cancelled Action run leaves the coordination lock held until TTL expiry, blocking re-runs #1835

Closed

Description

@marcusrbrown

Problem

A cancelled Action run leaves the per-repo coordination lock (coordination/<owner>/<repo>/locks/action.json) held until its 15-minute TTL runs out. Every attempt to re-run during that window skips, because the action sees "Another Fro Bot Action run is active for this repository", even though no run is active.

What happened

Observed on fro-bot/agent@83ed683 (v0.119.2), in a scheduled run on marcusrbrown/infra:

  1. Attempt 1 acquired the lock as action:<runId>:1 and renewed the lease every 30 s. I cancelled it about two minutes later to switch models. The runner killed the main step before runCleanup's finally block reached releaseLock (src/harness/phases/cleanup.ts). The last lease renewal was the last write to the lock.
  2. The post: step then ran its not-persisted retry path. It saved the cache and synced sessions/opencode.db to the object store. The post hook has no handle on the lock (src/harness/post.ts), so the lock stayed held.
  3. Attempts 2 and 3 of the same run got a new holder ID each (action:<runId>:2, action:<runId>:3). Both read the lock with the etag from attempt 1's last renewal, logged lock-held-by-other-surface … heldBy: action:<runId>:1, reason: coordination lease held, and skipped without running the agent. Both attempts failed in the next step, which depends on the agent run.

The run became usable only after the lease expired. At that point reclaimableWithoutConfirmation (src/harness/phases/acquire-lock.ts) allows taking over the expired Action-surface lease.

Why the lock should be releasable

When attempt N of a run starts, every earlier attempt of that run has finished. GitHub starts a re-run only after the previous attempt completes, post steps included. The runner also kills the job's leftover processes when the job ends: attempt 1's log shows Terminate orphan process: pid (…) (opencode). A lock held by action:<runId>:<M> with M < N can no longer have a live writer.

Proposed fixes

Either fix alone would have prevented this:

  • Release in the post hook. The main step records its holder ID with core.saveState after acquiring the lock, and clears that record after a successful release. When the post hook finds the record still set, it deletes the lock with a conditional delete that only succeeds while holder_id still matches. This covers a cancellation or crash between acquiring and releasing.
  • Same-run takeover. In acquire-lock, a lease held by an earlier attempt of the same run (parseActionHolderRunId(holder) === runId and holder attempt < current attempt) can be taken over without waiting for expiry. The existing etag precondition keeps the takeover safe.

The second fix also covers runners that never get to run the post step, for example when the runner is lost.

Activity

  1. fro-bot commented on Oct 15, 2026

    @fro-bot
    Owner

    Triage: confirmed lock-recovery bug

    Cancellation can leave the Action coordination lease behind, blocking immediate reruns for up to 15 minutes after the last successful renewal. The reported behavior is consistent with current main.

    Evidence

    I reproduced the lock behavior locally using the real runtime acquisition/renewal functions and an in-memory conditional-write store:

    attempt 1: acquired and renewed; release omitted to simulate interrupted cleanup
    attempt 2: active-holder, heldBy=action:1234:1
    attempt 3: active-holder, heldBy=action:1234:1
    after expiry: acquired
    

    This confirms the lock-state failure; I did not perform a live Actions cancellation or inspect the original run logs. The existing runtime lock, Action acquisition, and post-hook suites also pass: 114 tests across 3 files, indicating missing regression coverage.

    Root cause

    • cleanup.ts:445–477 releases in finally, using the latest renewal ETag. That handles ordinary exceptions, but cannot guarantee release when the process is forcibly terminated.
    • post.ts retries persistence but has no lock-release path or saved lock-ownership state.
    • lock.ts:186–208 rejects every unexpired holder before evaluating reclaimableWithoutConfirmation. The Action holder includes the attempt number, but no earlier-attempt exception exists.

    Important implementation detail: broadening the existing reclaim predicate alone will not fix immediate reruns—it is only consulted after expiry.

    Recommended next steps

    1. Prioritize same-run, earlier-attempt takeover. Add an explicit pre-expiry recovery path restricted to GitHub-surface Action holders with the same run ID and a strictly lower, valid attempt number. Replace using the observed ETag; a concurrent renewal/replacement must yield a conflict. Preserve exclusion for other runs, equal/newer attempts, and malformed holders.
    2. Add post-hook release as complementary recovery. Save ownership immediately after acquisition. After any allowed post persistence finishes, read the current Action lock, verify the saved holder still owns it, and conditionally delete using that read’s ETag. Clear saved ownership after successful main-step release. Release must also run when persistence is skipped or throws; it must never delete a successor’s lock. Saving only the initial ETag is insufficient because renewal changes it.
    3. Cover both recovery paths: unexpired earlier-attempt takeover; rejection of unrelated/equal/newer holders; ETag conflicts; cancellation-state post release; changed-holder protection; and post persistence failure/skip branches. Then validate with one controlled cancel-and-rerun workflow.

    Until fixed, wait for the lease to expire before rerunning. The lock protects the shared S3 session database, so weakening repository-wide exclusion is not an appropriate workaround.


    Run Summary
    Field Value
    Event issues
    Repository fro-bot/agent
    Run ID 38681226825
    Cache hit
    Session ses_ec231ef27ffecY75ZFtOomxctt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions