Skip to content

feat: add a metrics command to the HITL REPL - #290

Closed
BrennanTM wants to merge 1 commit into
lanl:mainfrom
BrennanTM:hitl-metrics-command
Closed

BrennanTM wants to merge 1 commit into
lanl:mainfrom
BrennanTM:hitl-metrics-command

Conversation

@BrennanTM

Copy link
Copy Markdown
Contributor

Summary

Implements the metrics REPL command sketched by @luiarthur in #58: typing metrics in the HITL session renders the session timing and cost summary (per-node, per-tool, and per-LLM timing tables plus cost by model) for the current thread.

Two simplifications relative to the original sketch, both consequences of how the CLI has evolved since:

  • Metrics are enabled by default now, so no opt-in plumbing is needed.
  • Every REPL agent is instantiated with the session's shared thread id, so the session rollup already aggregates all agents on the thread and lists them in the summary. That removes the need for per-agent thread ids and last-agent tracking, and metrics therefore takes no argument.

Before any agent has run, the command prints the existing "No session data" panel rather than erroring. That panel is also what the original report ran into: at the time, metrics were off by default and the REPL gave each agent its own suffixed thread id, so no rollup existed under the session's thread.

Testing

  • New test in tests/cli/test_hitl.py (using the existing stubbed-dependency helper) verifies the command renders the summary for the session's thread.
  • Manually exercised both paths in a stubbed REPL: the no-data panel, and the full summary with a seeded rollup.
  • tests/cli/ passes apart from the pre-existing test_example_config_smoke failure, which requires OPENAI_API_KEY and is unrelated. ruff check and ruff format --check are clean on both changed files.

Closes #58

Adds `metrics` to the REPL, rendering the session timing and cost
summary (per-node, per-tool, and per-LLM timing plus cost by model)
for the session's thread via render_session_summary. Prints the
existing "No session data" panel gracefully when no agent has run yet.

Compared to the approach sketched in lanl#58, no per-agent thread ids or
last-agent tracking are needed anymore: every REPL agent shares the
session thread id, so the rollup already aggregates the whole session
and lists the agents involved.

Closes lanl#58
@BrennanTM BrennanTM mentioned this pull request Aug 2, 2026
@mikegros

mikegros commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

I will happily check this out and probably get it merged soon. I think there is some extra complication in this since the move to persistent agents/centralized checkpointing, but I'll wait to ramble more about that until after I've had a chance to take a look.

@mikegros

mikegros commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Sorry, got distracted from this. One thing I'll note is that I think it works well for non-persistent sessions but doesn't play nice with persistent agents.

I think there's a pretty easy extension that would allow you to see both the current session cost and all-time cost with a persistent agent, but the second might need a little extra metadata included in the ursa_metrics folder. Just to note, for persistent agents the ursa_metrics files get put in the `.cache/ursa//agents/ directory. Rather than the workspace as happens with non-persistent sessions.

@BrennanTM

Copy link
Copy Markdown
Contributor Author

No worries, and thanks for the pointer on the den layout. That matches what I see in the code: persistent agents write telemetry to <den>/ursa_metrics with the den resolving under .cache/ursa/<group>/agents/<agent name>, while ephemeral sessions use the workspace.

The extension looks straightforward from here. The current command stays as the session view (in-process rollup, unchanged). For a named agent, metrics would additionally render an all-time panel, and the aggregation for it is already there in SessionRollup.ingest, the same rollup the session view renders: ingest every JSON in the den's ursa_metrics into a fresh rollup and render it, with a small refactor so render_session_summary can render a passed-in rollup, and with the elapsed-window line dropped (with everything on thread "ursa" it would span the agent's whole life). Cost comes from each file's costs block. One honesty caveat: files are priced at save time, so runs with unpriced or usage-less models sit at $0. The event_sources counters are in each file, so the panel can say "N events unpriced" rather than silently undercounting.

On the extra metadata: for a den-scoped scan the path already identifies the agent, so v1 doesn't strictly need it, but I would stamp the group and configured agent name into the payload context alongside the class name for provenance, kept optional on read since all existing files lack them. The one thing stamping doesn't solve: there is no session marker in the files (run_id is per-run, thread_id is always "ursa"), so if "per-session cost, historically" is part of what you want, that is the metadata I would add.

One gap worth flagging: import-agent db/checkpointer.db brings over the checkpointer but not the workspace's old ursa_metrics, so for migrated agents the all-time view would start at migration day. I would label the panel "recorded runs for this agent", and separately import-agent could learn to carry the metrics folder along.

My lean: merge this PR as the session view and do the extension as a follow-up, since it touches timing.py (and base.py if the metadata stamping goes in) and has a couple of small design calls in it, so it seems worth its own review. Happy to push it onto this branch instead if you'd rather take it in one go.

@mikegros

mikegros commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

I think the new TUI removes the need for this so I will close for now. We can readdress if there is a clear value.

@mikegros mikegros closed this Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

render_session_summary in HITL mode

2 participants