retryFailedStep ignores factor/minTimeout/maxTimeout with the Playwright helper (and applies them nondeterministically under Bun) #5723

What are you trying to achieve?

Configure the backoff of retryFailedStep (factor, minTimeout, maxTimeout) with the Playwright helper, and get the same retry timing on every run and runtime.

What do you get instead?

With the Playwright helper, the plugin's timing options are:

• ignored under Node: the backoff always uses promise-retry's defaults (factor: 2);
• applied only to some steps under Bun: the backoff alternates between the plugin's options and the defaults, so it depends on how many recorder tasks ran before the failing step.

Only retries is always right.

Minimal repro (a step that fails instantly, so the duration is pure backoff):

// codecept.conf.js
export const config = {
  tests: './*_test.js',
  output: './output',
  helpers: { Playwright: { url: 'about:blank', show: false, browser: 'chromium' } },
  plugins: { retryFailedStep: { enabled: true, retries: 4 } }, // default factor 1.5
}

// retry_test.js
Feature('retry')
Scenario('1 pass', ({ I }) => { I.amOnPage('about:blank') })
Scenario('2 fail', ({ I }) => { I.seeElement('#nope') })
Scenario('3 fail', ({ I }) => { I.seeElement('#nope') })

DEBUG=codeceptjs:recorder shows the Retry opts alternating between the plugin config and {"retries":4} on consecutive tasks under Bun.

Cause

Two retry configs are active in each test:

1. the plugin's (factor: 1.5, minTimeout, maxTimeout, …), pushed in test.before;
2. the Playwright helper's own, pushed in _before() (lib/helper/Playwright.js, since 3.7.0):

   this._retryConfig = {
     retries: test?.opts?.conditionalRetries || 3,
     when: err => /* only errors containing 'context' */,
   }
   recorder.retry(this._retryConfig)

   It copies only the retry count from the plugin, via conditionalRetries. Without factor/minTimeout/maxTimeout it falls back to promise-retry's defaults.

recorder.add() picks one config per task:

const retryOpts = this.retries
  .sort((r1, r2) => r1.when && !r2.when)
  .slice(-1)
  .pop()

• The comparator returns a boolean. Per the spec that isn't a consistent comparator, so the order is implementation-defined. V8 keeps the order, so the helper's config (pushed last) always wins. JavaScriptCore (Bun, also Safari/WebKit) swaps the two elements on every call, and .sort() mutates this.retries in place, so the winner alternates from task to task. WebKit closed the same pattern as not a bug: https://bugs.webkit.org/show_bug.cgi?id=289223
• Even with a deterministic sort, the timing comes from the helper's config, not from the rule whose when actually matched the error. For an element-not-found error only the plugin's when matches, yet the helper's defaults set the backoff.

Suggested fix

• Use a numeric comparator, e.g. (r1.when ? 0 : 1) - (r2.when ? 0 : 1), and avoid sorting this.retries in place.
• Take the promise-retry options from the retry rule that matched the error, or have the Playwright helper inherit all of the plugin's retry options, not only retries.

Workaround: set factor: 2 (and leave minTimeout/maxTimeout at their defaults) in the plugin config, so both configs are identical.

Details

• CodeceptJS version: 4.2.0 (same code since 3.7.0)
• NodeJS Version: 24.18.0 / Bun 1.4.2
• Operating System: Linux
• Playwright helper

Created by AI agent (Claude Code)