Tier 1 of the plan in #259

[BrennanTM]

Summary

This is Tier 1 of the plan laid out in #259: repair the existing OTLP trace export so it is correct, observable, and documented, and make the exported trace carry the run's real timeline. There is no new user surface beyond the existing save_otel flag and the otel extra, and the default path (export off) behaves exactly as before. Every defect from the audit was reproduced first, pinned by a pre-registered failing test, and the tests flipped green with no test edits after each repair.

What was broken and what changed

Audit item	Before	Now	Pinned by
Silent no-op without the extra	save_otel=True did nothing, invisibly	Warning with the exact install hint plus a structured no-op result	Subprocess import-blocker test and an in-process variant
Global provider misrouting	Every export called set_tracer_provider; the SDK honors only the first, so later exports landed on the first endpoint ([2, 0] repro in #259) and each call leaked a provider and thread	A private provider per export call; the code never touches the OTel global, so co-tenants such as chromadb and MLflow are unaffected	Two-endpoint routing test asserting [1, 1], plus an atexit-cleanliness pin
Nothing flushed	Spans waited on the batch timer or atexit; hard exits lost telemetry	Synchronous force_flush (10 s timeout) and shutdown inside the call, in try/finally	Span-exported-before-return test
Headers unusable as documented	The str-annotated otel_headers raised ValueError at exporter construction	A mapping or an env-style "k=v,k2=v2" string (parsed with the SDK's own liberal parser); anything else raises a ValueError that names otel_headers	Three header tests including a non-dict Mapping
Dead members	otel_metrics accepted and never read; Telemetry.otel_endpoint never read	otel_metrics now warns DeprecationWarning (Tier 2 supersedes it); the field joined the endpoint resolution order	DeprecationWarning test; field-honored test
Accidental endpoint resolution	Env vars worked only because the field was ignored	Explicit order: call parameter, then field, then the standard OTEL_* variables, then the SDK default; characterization tests pin that the pre-existing env behavior survived	Four precedence tests
No outcome signal, dishonest annotation	_save_otel claimed -> str and returned its argument; success and failure were both invisible	Structured {ok, endpoint, span_count, reason} result, with success and failure logged on ursa.observability.otel	Structured-result and caplog tests
Near-empty span	One span of about 0.03 ms of export-call time	The run's real trace (below)	Six span-tree tests

The exported trace now

One trace per export: a root span named with the run id spanning the run's real started_at to ended_at and carrying the time-breakdown and token-total attributes, plus one SpanKind.CLIENT child span per recorded LLM call at the call's real wall-clock interval. Children are named chat <model> and carry gen_ai.operation.name, gen_ai.request.model, gen_ai.provider.name, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, ursa.langgraph.node, and ursa.langgraph.step when the recorder captured them; the model and provider come from the ls_model_name and ls_provider keys that LangChain callbacks attach. Failed calls export with ERROR status and the error text as the status description. Malformed events cannot raise out of the export: assembly is guarded, the root always ends, and a partial trace with a warning beats a crashed finally.

Attribute names are string literals following the OpenTelemetry GenAI semantic conventions. Those conventions are Development status and recently moved to the semantic-conventions-genai repository, so names may still evolve upstream; the docs page and the docstring cite the new home.

The six questions from #259, answered with reversible defaults

Rather than blocking Tier 1 on the thread, each open question got a documented default chosen to be cheap to reverse:

Question	Default in this PR	Reversal cost
1. Which signal first, and which protocol	Traces first, OTLP/HTTP only: llm_events already carries real timestamps, and MLflow ingestion is HTTP-only. Standard OTEL_* env vars are honored	Tier 2 adds metrics on the same Resource; a gRPC exporter could be a separate extra
2. Which tier to start with	Tier 1 only	Not applicable
3. Global or private provider	Private per-call provider, never set_tracer_provider, per the OTel library guidelines cited in #259	None expected; Tier 2's MeterProvider follows the same pattern
4. Flush semantics and unreachable collectors	Synchronous flush with a 10 s timeout inside render(). ok means the spans were handed to the exporter and flushed; the SDK logs delivery failures. Export never raises into the agent, and an unreachable collector is bounded by the flush and shutdown timeouts and reported in the result and log	Timeout is one constant
5. Attribute constraints	Token counts, model and provider names, and node and step labels only. No prompt or completion content anywhere in Tier 1; content capture, if ever wanted, would be a separate opt-in following the semconv mechanism	Additive
6. CLI or REPL toggle	Deferred; the per-invoke flag is the only switch. A CLI toggle pairs naturally with the metrics command from #290 once Tier 2 lands	Additive

Ecosystem alignment

Exporting a completed run after the fact is explicitly supported by the OTel Trace API, which defines span creation and end with explicit timestamps from the past. The resulting shape matches how the ecosystem structures LLM traces: MLflow's opt-in GenAI semconv export mode translates its spans into the same gen_ai.* attributes per LLM call, and OpenLLMetry migrated its attribute names to exactly the names used here (gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens). OpenInference uses its own namespace but deliberately leaves span naming unregulated, so nothing here conflicts with it. The one place URSA diverges from ecosystem practice is the root span name (the run id), which is high-cardinality; the GenAI agent-spans convention (invoke_agent <agent name> with gen_ai.agent.name and conversation id attributes) is the natural Tier 2 upgrade and pairs with metrics on the same Resource.

Packaging change that needs your sign-off

The otel extra now installs opentelemetry-exporter-otlp-proto-http>=1.39.0 instead of the opentelemetry-exporter-otlp umbrella. URSA imports only opentelemetry.exporter.otlp.proto.http, and the umbrella additionally pulled in the unused gRPC exporter and grpcio wheels. It is a one-line revert if you prefer the umbrella. The accompanying uv.lock refresh was generated with a current uv (lockfile revision 3), which also normalizes some unrelated marker representation; I verified with uv export that the platform exclusions (for example the nvidia wheels staying linux-only) are unchanged. Related and worth weighing together: when the endpoint resolves through the SDK's own env or default path, the reported endpoint string is read from the exporter's private _endpoint attribute, which is stable across the pinned floor (verified at 1.44.0) but is private API.

Tests and CI

25 tests: 24 in tests/observability/test_otel_export.py plus a DeprecationWarning test in tests/agents/test_otel_kwargs.py. Method throughout: tests were written first and observed red before each fix. The two core repair waves committed their red tests separately beforehand (4960641 before 36b8af8, 596a2bf before 8005f55) and flipped green with zero test edits; the follow-up fixes carry their new red tests in the same commit, and no earlier test's body was ever edited. The load-bearing behaviors were mutation-checked (flushing, per-call provider routing, endpoint precedence including parameter over field, real timestamps, child parenting, live-metadata reads, the assembly guard, atexit cleanliness). The two subprocess tests total about 0.6 s.

Full suite locally: 362 passed; the only non-passes are pre-existing environment ones (the OPENAI_API_KEY smoke failure and three dashboard import errors without fastapi). CI installs --all-extras, so the battery runs on all three OS images; the new code and tests use nothing beyond Python 3.11, create no files or directories, and hold no handles that would trouble Windows.

Docs

New reference page, Reference then OpenTelemetry export: install and opt-in, the endpoint resolution table, header rules, collector and MLflow examples, the exported span shape, and a semantics-and-limits section that states plainly what ok does and does not promise, the re-export behavior, the unclamped-timestamps caveat, and the current error.type omission. The mkdocs build was validated locally; the site workflow publishes from main after merge.

Non-goals here (Tier 2 and beyond)

Metrics instruments; live node and tool spans (their records carry no timestamps today, so faithful spans need live instrumentation via the existing parent_run_id bookkeeping); the invoke_agent root rename; a CLI toggle; any content capture; error.type on failed children. The per-run console rendering gap that hid the old confirmation line predates this work, still hides the repaired one, and is filed separately as #304.

Changelog bullet for the next release notes

Following the #291 convention of compiling the changelog at release time, here is a paste-ready bullet:

### Observability

- Repaired the opt-in OpenTelemetry export (#259, Tier 1): exported traces
  now carry the run's real timeline with one child span per LLM call
  following the GenAI conventions, export uses a private per-call tracer
  provider with a synchronous flush, endpoints and headers resolve in a
  documented order with standard OTEL_* environment support, a missing
  otel extra now warns with an install hint, and the otel extra was
  narrowed to the HTTP exporter. Adds tests and a usage reference page.