Skip to content

Validate the Kubeflow integration with Identity and Istio Ambient using the single domain setup #1451

Closed
Closed
Validate the Kubeflow integration with Identity and Istio Ambient using the single domain setup#1451
Labels
enhancementNew feature or request

Description

@NohaIhab
NohaIhab
opened on Jun 30, 2026

Context

We need to validate and document how to deploy and integrate with the Canonical Identity stack and the Istio Ambient charms using the single domain setup with traefik

What needs to get done

  1. Deploy CKF + Istio Ambient + Identity
  2. Validate the UI access and programmatic access through a single domain (path-based)
  3. Document the steps for the setup and note down any workarounds

Definition of Done

We have a documentation of how to do the Identity integration with single path-based URL

Activity

  1. NohaIhab
    added
    enhancementNew feature or request
    on Jun 30, 2026

  2. syncronize-issues-to-jira commented on Jun 30, 2026

    @syncronize-issues-to-jira
    syncronize-issues-to-jiraBot
    on Jun 30, 2026 – with Syncronize issues to Jira

    Thank you for reporting your feedback to us!

    The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-8825.

    This message was autogenerated

  3. NohaIhab commented on Jul 21, 2026

    @NohaIhab
    NohaIhab
    on Jul 21, 2026
    ContributorAuthor

    Path-based single-domain setup: Traefik in front of two Istio ingress gateways

    Note: this comment was written with the help of AI

    This comment captures the work and manual effort required to validate a path-based, single-domain ingress setup where a single Traefik instance sits in front of two Istio ingress gateway charms (istio-ingress-k8s-ui and istio-ingress-k8s-m2m).

    The setup builds on the ambient IAM solution in charmed-kubeflow-solutions (feat/iam-integration).

    Overview

    The goal is to expose Kubeflow behind a single hostname (kubeflow.com) using path-based routing through Traefik, which forwards traffic to the Istio ingress gateways. Getting this working end-to-end required a mix of Juju integrations and several manual post-deployment steps (Traefik config, DNS, certificate trust, an AuthorizationPolicy edit, and a Pebble plan tweak).

    Topology

    flowchart LR
    client([Client / browser]) -->|https://kubeflow.com| traefik[Traefik]
    traefik -->|/kubeflow-istio-ingress-k8s-ui + X-Forwarded-Prefix| uigw[istio-ingress-k8s-ui]
    traefik -->|/kubeflow-istio-ingress-k8s-m2m + X-Forwarded-Prefix| m2mgw[istio-ingress-k8s-m2m]
    uigw --> kf[Kubeflow workloads / webapps]
    m2mgw --> kf
    Loading

    Models involved:

    • iam-core — hosts Traefik.
    • kubeflow — hosts the Istio ingress gateway charms and Kubeflow workloads.

    1. Juju integrations

    Offer Traefik's ingress endpoint from the iam-core model, consume it in the kubeflow model, and integrate both Istio ingress gateway charms as upstream ingress consumers.

    juju offer iam-core.traefik:ingress traefik-ingress
juju consume -m kubeflow admin/iam-core.traefik-ingress
juju integrate -m kubeflow istio-ingress-k8s-ui:upstream-ingress  traefik-ingress
juju integrate -m kubeflow istio-ingress-k8s-m2m:upstream-ingress traefik-ingress

    2. Traefik configuration (manual)

    Both ingress gateways are fronted by Traefik through a single dynamic-configuration file that Traefik loads via its file provider (which watches /opt/traefik/juju/ and applies changes without a restart). Each gateway gets its own middleware (to inject the correct X-Forwarded-Prefix), router, and backend service:

    • UI gateway (istio-ingress-k8s-ui) — served under /kubeflow-istio-ingress-k8s-ui.
    • m2m gateway (istio-ingress-k8s-m2m) — served under /kubeflow-istio-ingress-k8s-m2m.

    The path prefix for each gateway is not arbitrary. Over the ingress relation the requirer (the Istio ingress charm) publishes its model and name, and Traefik derives the route path prefix from them as <model>-<name>. For these charms in the kubeflow model that resolves to /kubeflow-istio-ingress-k8s-ui and /kubeflow-istio-ingress-k8s-m2m. The manual config below therefore reuses those exact prefixes; it only adds the X-Forwarded-Prefix header injection that the auto-generated routers don't provide.

    Manual step. This file is written directly into the running Traefik container. Adjust each service url to match your UI and m2m ingress gateway addresses (the IPs below are placeholders). Charm-native support for this configuration is requested in canonical/traefik-k8s-operator#751.

    kubectl exec -i -n iam-core traefik-0 -c traefik -- sh -c 'cat << "EOF" > /opt/traefik/juju/juju_kubeflow_ingress_routes.yaml
http:
  middlewares:
    kubeflow-ui-header:
      headers:
        customRequestHeaders:
          X-Forwarded-Prefix: "/kubeflow-istio-ingress-k8s-ui"
    kubeflow-m2m-header:
      headers:
        customRequestHeaders:
          X-Forwarded-Prefix: "/kubeflow-istio-ingress-k8s-m2m"
  routers:
    kubeflow-ui-tls:
      entryPoints:
        - websecure
      rule: PathPrefix(`/kubeflow-istio-ingress-k8s-ui`)
      service: kubeflow-ui-service
      middlewares:
        - kubeflow-ui-header
      tls: {}
    kubeflow-m2m-tls:
      entryPoints:
        - websecure
      rule: PathPrefix(`/kubeflow-istio-ingress-k8s-m2m`)
      service: kubeflow-m2m-service
      middlewares:
        - kubeflow-m2m-header
      tls: {}
  services:
    kubeflow-ui-service:
      loadBalancer:
        servers:
          - url: https://10.64.140.42:443
    kubeflow-m2m-service:
      loadBalancer:
        servers:
          - url: https://10.64.140.43:443
EOF'

    3. DNS

    Point kubeflow.com at the Traefik LoadBalancer IP, both in-cluster (CoreDNS) and on the workstation (/etc/hosts).

    # configmap the coredns deployment actually mounts
CM=$(kubectl -n kube-system get deploy coredns \
    -o jsonpath='{.spec.template.spec.volumes[?(@.configMap)].configMap.name}')
echo "CoreDNS mounts: $CM"

# traefik LB IP
TRAEFIK_IP=$(kubectl -n iam-core get svc traefik-lb -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null \
    || kubectl -n iam-core get svc -l app.kubernetes.io/name=traefik-k8s -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')
echo "Traefik LB IP: $TRAEFIK_IP"

# set the hostname on traefik
juju config -m iam-core traefik external_hostname=kubeflow.com

# patch the mounted Corefile in one pass
kubectl -n kube-system get configmap "$CM" -o yaml \
  | TRAEFIK_IP="$TRAEFIK_IP" HOST=kubeflow.com python3 -c '
import sys, os, re, yaml
d  = yaml.safe_load(sys.stdin)
ip, host = os.environ["TRAEFIK_IP"], os.environ["HOST"]
cf = d["data"]["Corefile"]
if host in cf:
    cf = re.sub(r"[0-9.]+ " + re.escape(host), ip + " " + host, cf)
else:
    block = "    hosts {\n        %s %s\n        fallthrough\n    }\n    " % (ip, host)
    cf = cf.replace("kubernetes cluster.local", block + "kubernetes cluster.local", 1)
out = {"apiVersion":"v1","kind":"ConfigMap",
       "metadata":{"name":d["metadata"]["name"],"namespace":d["metadata"]["namespace"]},
       "data":{**d["data"], "Corefile":cf}}
print(yaml.safe_dump(out))
' | kubectl apply -f -

# restart coredns and confirm
kubectl -n kube-system rollout restart deployment/coredns
kubectl -n kube-system rollout status deployment/coredns
kubectl -n kube-system get configmap "$CM" -o jsonpath='{.data.Corefile}' | grep "$TRAEFIK_IP kubeflow.com"

# add the same mapping to your workstation's hosts (replace any existing kubeflow.com line first)
sudo sed -i '/[[:space:]]kubeflow\.com$/d' /etc/hosts
echo "$TRAEFIK_IP kubeflow.com" | sudo tee -a /etc/hosts

    Notes:

    • The script is idempotent: the Corefile patch updates an existing kubeflow.com mapping or inserts a new hosts { ... } block before the kubernetes cluster.local plugin line, and the /etc/hosts step strips any existing kubeflow.com line before appending the current one (so reruns and IP changes don't leave duplicate or stale entries).
    • In-cluster DNS is needed so services resolve kubeflow.com to Traefik; the /etc/hosts entry does the same for the workstation/browser.

    4. Certificates

    Traefik connects to the Istio gateways over HTTPS, so it must trust the Kubeflow gateway's CA. Offer the CA from the kubeflow model and add it to Traefik's trust store via the dedicated receive-ca-cert endpoint.

    # offer the kubeflow gateway CA
juju offer kubeflow.self-signed-certificates:send-ca-cert kubeflow-ca

# consume into iam-core (where traefik lives)
juju consume -m iam-core admin/kubeflow.kubeflow-ca

# add it to traefik's trust store (separate endpoint from :certificates)
juju integrate -m iam-core traefik:receive-ca-cert kubeflow-ca

    5. AuthorizationPolicy

    The ext-authz AuthorizationPolicy is charm-generated (name ext-authz-istio-ingress-k8s-ui, action: CUSTOM, provider.name, targetRefs, and the when: notValues: [Bearer *] clause). The static assets the browser fetches before login (/favicon.ico, /robots.txt) must be added to its notPaths — otherwise those pre-login requests are sent through ext-authz and the login fails.

    Patch the charm-created UI policy:

    kubectl -n kubeflow patch authorizationpolicy ext-authz-istio-ingress-k8s-ui --type=merge -p '
spec:
  rules:
  - to:
    - operation:
        notPaths:
        - /favicon.ico
        - /robots.txt
    when:
    - key: request.headers[authorization]
      notValues:
      - Bearer *
'

    Manual step / known limitation. The charm owns this resource, so the edit is overwritten on reconcile. Tracked in canonical/service-mesh#151.

    The m2m gateway needs no ext-authz — it is JWT-only, enforced by the charm's deny-without-jwt-<app> DENY policy.

    6. kubeflow-dashboard Pebble plan (HTTP header size)

    Symptom

    The browser returned a 431 Request Header Fields Too Large error. The Istio gateway logs pointed to an upstream issue on the kubeflow-dashboard workload rather than the gateway itself.

    Root cause & fix

    The dashboard runs on Node.js, whose default maximum HTTP header size was too small for the accumulated auth/proxy headers. Raising Node's --max-http-header-size (via NODE_OPTIONS) in the Pebble plan resolves it.

    This is Node.js behaviour, not a Kubeflow bug: Node returns 431 Request Header Fields Too Large once the header limit is exceeded (nodejs/node#25528), and the --max-http-header-size flag used here to raise that limit was added in nodejs/node#24811.

    charm/bin/pebble add --combine header-size /dev/stdin <<'EOF'
services:
  kubeflow-dashboard:
    override: merge
    environment:
      NODE_OPTIONS: '--max-http-header-size=32768'
EOF

    This increases Node's max header size to 32 KiB (default is 16 KiB).

    Charm enhancement issue opened. Making the max header size configurable through the charm (instead of the manual Pebble edit above) is tracked in canonical/kubeflow-dashboard-operator#338.

    Summary

    The work splits into two categories: expected setup steps that form the normal deployment configuration for this topology, and gaps that currently require manual intervention because charm-native support is missing.

    Expected setup steps

    Normal deployment configuration for this topology — not bugs or workarounds. Once the tracked gaps below are implemented, these are the steps that would form part of a user-facing setup guide.

    Area Action Charm-managed? Notes
    Ingress wiring Offer/consume/integrate Traefik ↔ Istio gateways Yes Can be defined declaratively in the Terraform solution
    Certificates Offer/consume/integrate CA into Traefik trust store Yes Uses receive-ca-cert; can be defined declaratively in the Terraform solution (covers both gateways if they share the CA)
    DNS Patch CoreDNS Corefile + /etc/hosts Partial external_hostname via Juju; Corefile is standard DNS config

    Gaps (tracked)

    Steps that currently require manual intervention because charm-native support is missing. Each has an issue open to close the gap.

    Area Action Charm-managed? Notes
    X-Forwarded-Prefix injection Write dynamic file into Traefik container to inject X-Forwarded-Prefix per route Partial The <model>-<name> path routing to each backend is already charm-provided; only the X-Forwarded-Prefix header (and a root / fallback) must be added manually — traefik-k8s-operator#751
    ext-authz notPaths bypass Patch the charm-created UI ext-authz policy to add the pre-login static-asset paths (/favicon.ico, /robots.txt) Partial These notPaths must be added manually and are overwritten on reconcile. m2m is JWT-only and needs no ext-authz — service-mesh#151
    Dashboard header size Pebble NODE_OPTIONS tweak No Fixes 431 errors — kubeflow-dashboard-operator#338

  4. MattiaSarti commented on Jul 22, 2026

    @MattiaSarti
    MattiaSarti
    on Jul 22, 2026

    Peer-reviewed - excellent work!

  5. NohaIhab commented on Jul 23, 2026

    @NohaIhab
    NohaIhab
    on Jul 23, 2026
    ContributorAuthor

    Toward a charm-native implementation

    The manual Traefik config is the main gap (traefik-k8s-operator#751). A couple of directions came out of discussion with the Service Mesh team for how the custom path prefixes, X-Forwarded-Prefix injection, and root (/) fallback could be provided with charms:

    • traefik-route relation. Traefik's traefik-route interface lets a requirer submit arbitrary dynamic config — routers, services, and middlewares — which is exactly what the manual file does, but declaratively over a relation. The idea would be to relate the Istio ingress gateway to Traefik over traefik-route.

      • Design consideration: traefik-route config applies globally to the whole Istio ingress application, so the config needs a home that is not the downstream Kubeflow charms — we don't want each workload charm supplying global routing/middleware config.

    • ingress-configurator + ingress. ingress-configurator is a small existing charm that acts as a dedicated place to hold ingress configuration, decoupled from the downstream workload charms. This is the favored implementation because it already exists.

    These are options to weigh when we prioritize charming this implementation — captured here so the design context isn't lost.

  6. NohaIhab
    closed this as completedon Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions