Open Telemetry #259

Activity

1. mikegros commented on Jun 13, 2026

   

   mikegros

   on Jun 13, 2026

   Collaborator

   More actions

   I think some is included already as an optional dependence. We can get it fully integrated though
2. jsta commented on Jun 15, 2026

   

   jsta

   on Jun 15, 2026

   Member

   More actions

   see #98
3. BrennanTM commented on Aug 2, 2026

   

   BrennanTM

   on Aug 2, 2026

   Contributor

   More actions

   I spent some time mapping what "fully integrated" could mean here, since partial support already exists. Below is where things stand, what the standards landscape now offers, and a tiered proposal, with questions at the end.

   What exists today

   #143 (thanks @jsta) answered #98 with an opt-in bridge: invoke(save_otel=True, ...) exports one summary span per run over OTLP/HTTP, and the #145 guard made the dependency optional. The design choices make sense for their era: MLflow ingests OTLP at /v1/traces on the tracking port (5000) and routes by an x-mlflow-experiment-id header, which is exactly the default endpoint and the headers parameter the bridge carries.

   Even when export works, the span itself is nearly information-free: it carries about 0.03 ms of export-call duration rather than the run's timeline, because the started_at/ended_at parsing is commented out (timing.py:1228-1229). Working through the code and confirming each point empirically (fresh-process runs against a captured exporter), the bridge has some real problems today:

   1. On a default install the OTLP/HTTP exporter is absent (only the gRPC exporter arrives transitively via chromadb), so save_otel=True silently does nothing. Success is also invisible: render() builds its confirmation line but never displays it, and _save_otel returns its endpoint argument, which is None on the default path even when export worked.
   2. _save_otel constructs a new SDK TracerProvider and calls trace.set_tracer_provider() on every export call (timing.py:1232). The SDK honors only the first set, so every later export is misrouted to the first call's exporter while the code reports the new endpoint as a success, and each call leaks a provider plus a batch-export daemon thread. I verified two exports to different endpoints land both spans on the first endpoint and none on the second ([2, 0]). Setting the global also collides with in-process co-tenants: chromadb's opt-in OTel sets the global provider, and MLflow's tracing SDK "automatically integrates with OpenTelemetry's default tracer provider". OTel's guidance is that instrumentation depends only on the API and the application decides how the SDK is configured (library guidelines, instrumentation docs).
   3. Nothing flushes: spans wait on the roughly 5 second batch timer or the atexit hook, so hard exits lose telemetry. The trace SDK spec defines ForceFlush for exactly this hazard class.
   4. otel_headers is typed str, but the exporter requires a mapping; a string raises ValueError at construction.
   5. otel_metrics on BaseAgent.__init__ (base.py:223) is accepted and never read, and Telemetry.otel_endpoint (timing.py:1049) is likewise dead code. When the endpoint is omitted, the exporter actually honors the standard OTEL_EXPORTER_OTLP_* env vars already.
   6. There is no CLI surface, no tests, and no usage docs (the sole mention is one word in the installation extras list).
   7. Found along the way, independent of OTel: on_llm_error unpacks a 4-element fallback tuple into 5 names (timing.py:771-773), so an LLM error arriving without a matching start raises ValueError. Happy to file that separately with a repro.

   Two structural facts shape what is possible. llm_events already stores t_start/t_end plus langgraph_node/langgraph_step for every LLM call, so true-time LLM child spans can be built from the saved payload today. Node and tool records are (name, ms, ok) aggregates with no timestamps, so faithful node and tool spans require opening spans live, and the hooks for that exist: the Per*Timer callbacks already do run_id-keyed begin/end bookkeeping and receive the parent_run_id needed for a real span tree (today it is only checked against None to split roots from children, never used for linkage).

   The standards picture

   OpenTelemetry now has GenAI semantic conventions (Development status, so names can still change): agent spans (invoke_agent, plan, execute_tool, invoke_workflow) and named metric instruments that line up almost one to one with what URSA already aggregates:

   URSA today	OTel GenAI instrument
   per-run wall/LLM/tool totals	gen_ai.invoke_agent.duration, .inference_calls, .tool_calls
   per-tool (name, ms, ok) buckets	gen_ai.execute_tool.duration (attr gen_ai.tool.name)
   workflow timings	gen_ai.workflow.duration
   token totals by model	gen_ai.client.token.usage

   The span fit is rough but honest: URSA's planning phase matches the plan span closely (the plan-generating LLM call "SHOULD be a child of the plan span"), while the reflect loop is an extension the convention does not model, and plan execution living in a separate agent means the convention's same-span sibling clause only holds with context propagation across the workflow.

   Ecosystem notes: an official opentelemetry-instrumentation-genai-langchain exists (development status, metrics support; declares langchain >= 0.3.21, which I have not verified against the 1.x rewrite URSA pins), Traceloop ships a mature third-party instrumentor that logs prompt/completion content to span attributes by default (the official posture is the inverse: content capture is opt-in via OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT), and the langsmith SDK already in URSA's tree can emit OTLP via LANGSMITH_OTEL_ENABLED with no new dependencies, though with no documented gen_ai shape and with LangSmith coupling. Any of these could substitute for parts of the work below, with tradeoffs I am happy to detail.

   Proposal, in tiers

   1. Repair the bridge (small): non-global provider, flush before return, real success and failure surfacing, mapping headers, delete the two dead members, an install-hint warning when the extra is missing, and optionally true-time LLM child spans from llm_events. Tests and docs included.
   2. Metrics export (small to medium): map the existing aggregates onto the named gen_ai instruments in the table above. Given this issue's wording, "a replacement from that 'metrics' stuff", this may be the literal ask.
   3. Live span tree (medium): open real spans at the existing callback hooks with parent_run_id linkage, giving faithful node, tool, and LLM traces.
   4. Full replacement (large): OTel as the only pipeline, console summaries derived from it. That is a direction decision I would not presume.

   Tiers 1 and 2 are independent and incremental; 3 layers on top of 1.

   Questions

   1. Which signal does Genesis need first: traces, metrics, or both? And the collector side: OTLP/HTTP or gRPC, and are standard OTEL_* env vars acceptable as the config surface? (If the target is MLflow, its ingestion is HTTP only.)
   2. Which tier or tiers should I start with?
   3. Should URSA use its own non-global provider to stay isolated from co-tenants like chromadb and MLflow, per the library guidance above?
   4. Per-run flush semantics: is a synchronous flush acceptable, and what should happen when the collector is unreachable?
   5. Attribute constraints: are token counts and node/model names acceptable to export? Any prompt/completion content capture ever, and if so, opt-in per the official mechanism?
   6. Would a CLI/REPL toggle be wanted (it would pair naturally with the metrics command from feat: add a metrics command to the HITL REPL #290)?

   If useful, I will start with tier 1 plus the tier 2 mapping behind the existing otel extra.
4. mikegros

   added a commit that references this issue on Sep 1, 2026

   Tier 1 of the plan in #259 (#305)

   ...

   9b4d8dc