[SarahLittlejohn]

Summary

Weekly flaky-test scan of the last 7 days of CI on master found one flaky test:

tests/verified-user/email-subscriptions.spec.ts:72 — Email Subscriptions - Location › complete subscription journey with authentication protection and accessibility

On the Nightly Tests run of 2026-08-09 this test failed on its first attempt and passed on Playwright's automatic retry — against the exact same commit (64a344d9) that had run clean on 2026-08-08 and 2026-08-10. Flake rate: 1/7 relevant nightly runs (~14%).

Root cause

The failure was:

Error: expect(page).toHaveURL(expected) failed
Expected pattern: /\/account-home/
Received string:  "https://localhost:8080/sign-in?error=auth_failed&lng=en"

exchangeCodeForToken() in libs/auth/src/cft-idam/token-client.ts makes a single, non-retried fetch call to the CFT IDAM (AAT) token endpoint. Any transient network blip or 5xx response from that external test environment throws, which the callback handler in apps/web/src/pages/(auth)/cft-login/return/index.ts catches and turns into a /sign-in?error=auth_failed redirect — the login journey fails outright instead of retrying, and the real cause (a one-off network/server hiccup) is masked.

This is external-network-reliance flakiness, not application logic — the redirect target of waitForURL in the e2e login helper is loose enough to also match the failure redirect, which is how the failure surfaced downstream in the test assertion rather than inside the helper itself.

Fix

Added a single bounded retry (2 attempts, 250ms backoff) around the token exchange call, retrying only on:

• network-level failures (fetch throwing — DNS, connection reset, timeout), and
• 5xx responses from IDAM (transient service errors)

4xx responses (e.g. an already-used, single-use authorization code) still fail immediately, since retrying those can never succeed and would just add latency to a genuine failure.

Verification

The sandboxed environment couldn't reach hmcts's private Azure DevOps package feed (@hmcts-cft/cloud-native-platform) needed for a full yarn install, so the vitest suite couldn't be run directly here. Instead the exact committed token-client.ts was executed directly (via tsx, type-only imports elided) against 7 mocked scenarios covering success, 4xx (no retry), 5xx (retry then succeed / retry then fail), and network failure (retry then succeed / retry then fail) — run 5 times consecutively with consistent, deterministic results. New unit tests covering the same scenarios were added to libs/auth/src/cft-idam/token-client.test.ts for CI to run with yarn test.

Other findings (not flaky, noted for visibility)

• 14 E2E tests fail on every nightly run in the window (CFT/Crime IDAM login flows, GOV.UK Notify email content, blob-ingestion API, etc.) — these never pass, so they don't meet the "passed and failed on the same code" definition of flaky. They look like tests depending on external service credentials/config not available in CI (e.g. COURTEL_API_URL is not set). Worth a separate ticket to fix or quarantine, but out of scope here.
• No flaky unit tests (Test Changed Packages / Vitest) were found on master pushes in the last 7 days — all green except one run where only the Terraform Apply jobs failed (infra, unrelated to tests).
• No prior week's flaky-test report exists to diff against, since this is the first run of this scan.

Test plan

• Added unit tests for retry-on-5xx, retry-on-network-failure, exhausted-retries, and no-retry-on-4xx behaviour in token-client.test.ts
• Verified the retry logic directly against the committed source (7 scenarios × 5 runs, deterministic)
• biome check clean on both changed files
• CI (yarn test) — could not run locally due to private package feed access; will be verified by the pipeline on this PR

---

Generated by Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Token exchange requests now retry automatically for temporary server and network failures.
  • Client-side errors fail immediately with clearer status and response details.
  • Exhausted retries now return a consistent error instead of failing unexpectedly.

• Tests

  • Added coverage for successful retries, non-retryable failures, retry exhaustion and network errors.

[coderabbitai]

Warning

Review limit reached

@SarahLittlejohn, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 50 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 814bf3d5-fd4d-4693-90ac-6f8a61097388

📥 Commits

Reviewing files that changed from the base of the PR and between 1845130 and 50a58c7.

📒 Files selected for processing (2)

• libs/auth/src/cft-idam/token-client.test.ts
• libs/auth/src/cft-idam/token-client.ts

📝 Walkthrough

Walkthrough

The token client now retries transient network and 5xx failures up to two attempts with a 250 ms delay. It fails immediately for 4xx responses. Tests cover successful retries and exhausted retries.

Changes

Token exchange retries

Layer / File(s)	Summary
Bounded token exchange retry flow libs/auth/src/cft-idam/token-client.ts	The client limits requests to two attempts and waits 250 ms between retries. It retries network and 5xx failures, returns successful JSON responses, and preserves final failure details.
Retry behaviour validation libs/auth/src/cft-idam/token-client.test.ts	Tests verify non-retryable 4xx failures, successful 5xx and network retries, retry exhaustion, fetch attempt counts, and resulting errors.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title mentions a flaky subscription journey and authentication, which relate to the fix, but it does not clearly identify the bounded token exchange retries.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/quirky-cray-foo8yc

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e04dcb78-726b-4047-a6fc-6cf9009c9825

📥 Commits

Reviewing files that changed from the base of the PR and between df61527 and 1845130.

📒 Files selected for processing (2)

• libs/auth/src/cft-idam/token-client.test.ts
• libs/auth/src/cft-idam/token-client.ts

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
96.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

Preview Deployment Successful 🚀

Your preview environment is ready:

• Api: https://cath-service-pr-930-api.dev.platform.hmcts.net
• Postgres: https://cath-service-pr-930-postgres.dev.platform.hmcts.net
• Web: https://cath-service-pr-930-web.dev.platform.hmcts.net

The environment will be automatically cleaned up when this PR is closed.

[github-actions]

🎭 Playwright E2E Test Results

82 tests   52 ✅  7m 30s ⏱️
31 suites  30 💤
 1 files     0 ❌

Results for commit 50a58c7.