[MaciejKaras]

Summary

After the Application Database password is changed, Ops Manager keeps the old password until it is rolled and returns 401 to the operator. The AppDB reconcile treated that 401 as a corrupted admin key and returned early with Ops Manager Failed, which skipped publishing the new password to the AppDB and rolling Ops Manager. Ops Manager then kept failing with the old password and never recovered.

The 401 is still reported in the Ops Manager status, but the AppDB reconcile now continues.

This also removes a dead err check after generateProcessHostnames. It re-used a stale err from an earlier, only-logged Ops Manager call and reported it as error getting current appdb statefulset hostnames, which is how a 401 from that call reached the early return. With the check gone, such a 401 no longer fails the reconcile; the early-return change covers 401s returned by later calls (for example registering the AppDB hosts).

Proof of Work

• unit: TestReconcileAppDB_DeploysAppDBWhenOpsManagerReturnsUnauthorized injects the 401 where the error is returned (GetHosts) and asserts the exact Ops Manager status (Failed with the admin-key message) and that the AppDB StatefulSet is still created; the reachable case asserts an empty status. With the early return restored, the 401 case fails. controllers/operator/... and controllers/om/... pass.
• e2e: targeted Evergreen patch for e2e_om_appdb_scram on the OM kind and multi-cluster variants passed (18 of 18 tasks, including build dependencies); a rerun with the final test passed too (18 of 18). The failure is a race, so one green run is supporting evidence only.
• evergreen: evergreen:e2e-light (≈147 tasks) — operator change; light runs e2e_om_appdb_scram on 8 variants. No OM-version-specific label, the change is version independent.

Checklist

• Have you linked a jira ticket and/or is the ticket in the title?
• Have you checked whether your jira ticket required DOCSP changes?
• Does your PR require running e2e tests?
• Does your PR produce user visible change?

[github-actions]

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.14.1 Release Notes

Bug Fixes

• MongoDBOpsManager: Fixed an issue where changing the Application Database password could leave Ops Manager in the Failed phase with The admin-key secret might be corrupted, because the new password was never applied to the Application Database.

[MaciejKaras]

evergreen retry