Skip to content

feat(e2e): preserve Kubernetes clusters for debugging #3675

Description

@danehans

User Story

As an OpenShell contributor debugging a Kubernetes E2E failure, I want to preserve the ephemeral cluster and test artifacts after the run so that I can inspect the failed state directly.

Problem Statement

The Kubernetes E2E wrapper always deletes the k3d cluster and temporary work directory from its exit trap. Its failure diagnostics capture common resources, but they cannot anticipate every investigation and remove the live state needed for follow-up commands.

Impact / Why This Matters

Contributors must reproduce a potentially slow or intermittent failure, modify the wrapper locally, or run the setup manually. This increases investigation time and can hide evidence that exists only immediately after a failure.

Proposed Design

Add an opt-in OPENSHELL_E2E_KUBE_PRESERVE_CLUSTER=1 mode for ephemeral clusters created by the E2E wrapper. When enabled, the wrapper preserves its cluster and work directory and prints their locations plus explicit cleanup commands. Existing cleanup remains the default, and externally supplied clusters are not reclassified as wrapper-owned resources.

Preserved work directories can contain short-lived test credentials, so the output and documentation must tell contributors to remove them after investigation.

Acceptance Criteria

  • OPENSHELL_E2E_KUBE_PRESERVE_CLUSTER=1 preserves an ephemeral cluster created by the wrapper and its work directory after success or failure.
  • The wrapper prints the cluster name, kubeconfig, work directory, and explicit commands to delete both preserved resources.
  • Default behavior continues to delete wrapper-created clusters and temporary files.
  • The option does not change ownership or cleanup behavior for a caller-supplied Kubernetes context.
  • Invalid option values fail before cluster creation.
  • Contributor documentation describes the option and the temporary credential cleanup requirement.

Alternatives Considered

Editing the exit trap locally is error-prone and not reproducible. Reusing a manually managed cluster helps some investigations but does not preserve the exact state produced by the standard ephemeral E2E workflow.

Agent Investigation

The cleanup trap in e2e/with-kube-gateway.sh collects a fixed diagnostic bundle, uninstalls fixtures, deletes a wrapper-created k3d cluster, and removes its temporary work directory. An early, opt-in preservation branch can retain the live environment without changing default behavior.

Checklist

  • I've reviewed existing issues and the architecture docs
  • This is a design proposal, not a "please build this" request

Activity

  1. danehans commented on Sep 24, 2026

    @danehans
    ContributorAuthor

    /assign

  2. danehans commented on Sep 24, 2026

    @danehans
    ContributorAuthor

    🏗️ build-plan

    Implementation Plan

    Issue type: feat
    Complexity: Low
    Confidence: High — clear path

    Summary

    Add an opt-in preservation mode for wrapper-created Kubernetes E2E clusters while retaining default and caller-owned-context cleanup behavior. Validate the option before creating resources and print safe inspection and cleanup guidance for preserved credentials and state.

    Scope

    • e2e/with-kube-gateway.sh: validate the option, preserve only wrapper-owned clusters and workdirs, retain failure diagnostics, and print inspection and cleanup commands.
    • tasks/scripts/test-kubernetes-e2e-preserve.sh: cover validation, preservation, default cleanup, and caller-owned contexts with deterministic command stubs.
    • tasks/test.toml: include the focused test in the standard test suite.
    • .agents/skills/helm-dev-environment/SKILL.md: document ownership semantics, retained credentials, inspection, and cleanup.

    Implementation Steps

    1. Validate and normalize OPENSHELL_E2E_KUBE_PRESERVE_CLUSTER before creating the work directory or cluster.
    2. Preserve wrapper-owned ephemeral state after failure diagnostics but before destructive cleanup.
    3. Print cluster, context, kubeconfig, workdir, inspection command, both cleanup commands, and a temporary-credential warning.
    4. Add deterministic regression tests for preserved, default, invalid, and external-context behavior.

    Test Plan

    • Unit/focused shell tests: Exercise wrapper lifecycle behavior through stubbed cluster commands.
    • Integration: bash -n, focused task, and mise run pre-commit.
    • E2E: Run the focused Kubernetes wrapper lifecycle path without requiring a live cluster; the change controls the harness lifecycle rather than OpenShell runtime behavior.

    Risks & Open Questions

    • Preserve the wrapped command's exit status and safely quote paths that may contain spaces.
    • Preserve only wrapper-created clusters; never reinterpret caller-owned contexts.

    Documentation Impact

    Contributor workflow documentation only; no published product or gateway configuration documentation changes.


    Revision 1 — initial plan

  3. danehans commented on Sep 24, 2026

    @danehans
    ContributorAuthor

    🏗️ build-from-issue-agent

    Implementation Complete

    PR: #3680

    What was built

    Added opt-in preservation for wrapper-created Kubernetes E2E clusters and work directories, with ownership-safe cleanup behavior and explicit credential warnings.

    Tests

    • Focused shell coverage: invalid input, success/failure preservation, default cleanup, and caller-owned contexts
    • mise run pre-commit
    • Live ephemeral k3d failure-path validation

    Docs updated

    • .agents/skills/helm-dev-environment/SKILL.md

    The issue will auto-close when the PR is merged.

  4. added 2 commits that reference this issue on Sep 24, 2026
    0bff5c4
    34854bd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions