Repository navigation
Conversation
Adds `metrics` to the REPL, rendering the session timing and cost summary (per-node, per-tool, and per-LLM timing plus cost by model) for the session's thread via render_session_summary. Prints the existing "No session data" panel gracefully when no agent has run yet. Compared to the approach sketched in lanl#58, no per-agent thread ids or last-agent tracking are needed anymore: every REPL agent shares the session thread id, so the rollup already aggregates the whole session and lists the agents involved. Closes lanl#58
|
I will happily check this out and probably get it merged soon. I think there is some extra complication in this since the move to persistent agents/centralized checkpointing, but I'll wait to ramble more about that until after I've had a chance to take a look. |
|
Sorry, got distracted from this. One thing I'll note is that I think it works well for non-persistent sessions but doesn't play nice with persistent agents. I think there's a pretty easy extension that would allow you to see both the current session cost and all-time cost with a persistent agent, but the second might need a little extra metadata included in the ursa_metrics folder. Just to note, for persistent agents the ursa_metrics files get put in the `.cache/ursa//agents/ directory. Rather than the workspace as happens with non-persistent sessions. |
|
No worries, and thanks for the pointer on the den layout. That matches what I see in the code: persistent agents write telemetry to The extension looks straightforward from here. The current command stays as the session view (in-process rollup, unchanged). For a named agent, On the extra metadata: for a den-scoped scan the path already identifies the agent, so v1 doesn't strictly need it, but I would stamp the group and configured agent name into the payload context alongside the class name for provenance, kept optional on read since all existing files lack them. The one thing stamping doesn't solve: there is no session marker in the files (run_id is per-run, thread_id is always "ursa"), so if "per-session cost, historically" is part of what you want, that is the metadata I would add. One gap worth flagging: My lean: merge this PR as the session view and do the extension as a follow-up, since it touches timing.py (and base.py if the metadata stamping goes in) and has a couple of small design calls in it, so it seems worth its own review. Happy to push it onto this branch instead if you'd rather take it in one go. |
|
I think the new TUI removes the need for this so I will close for now. We can readdress if there is a clear value. |
Summary
Implements the
metricsREPL command sketched by @luiarthur in #58: typingmetricsin the HITL session renders the session timing and cost summary (per-node, per-tool, and per-LLM timing tables plus cost by model) for the current thread.Two simplifications relative to the original sketch, both consequences of how the CLI has evolved since:
metricstherefore takes no argument.Before any agent has run, the command prints the existing "No session data" panel rather than erroring. That panel is also what the original report ran into: at the time, metrics were off by default and the REPL gave each agent its own suffixed thread id, so no rollup existed under the session's thread.
Testing
tests/cli/test_hitl.py(using the existing stubbed-dependency helper) verifies the command renders the summary for the session's thread.tests/cli/passes apart from the pre-existingtest_example_config_smokefailure, which requiresOPENAI_API_KEYand is unrelated.ruff checkandruff format --checkare clean on both changed files.Closes #58