Repository navigation
batch_evaluation fails on self-hosted v4 events_only deployments (uses unavailable v3 read endpoints) #1861
Description
Activity
Small correction: the original report referenced
BatchEvaluationRunner._fetch_items, which is a wrong name. The actual method name is_fetch_batch_with_retry(langfuse/batch_evaluation.py, line 1139 in v4.14.4), I've edited the issue bodyReacted by Trueman LamRepro and workaround sketch confirm the same root cause on a self-hosted v4 events_only stack —
GET /api/public/v3/tracesand the v3 observations path return events_only errors, while v2 observations works. Routing batch_evaluation throughGET /api/public/v2/observationsis the natural fix.One open question that would change the PR shape: would you prefer the default switch (the v3 read path is deprecated with EOL 2026-11-16 anyway) or a flag-gated rollout behind
LANGFUSE_USE_V2_OBSERVATIONS_API? Happy to align to whatever your usual approach is here.I have a fix + regression tests ready. Will send the PR within ~24h once direction is set; if you'd rather I hold until a maintainer pings, just say so.
Environment
docker.io/langfuse/langfuse:4), default write modeevents_onlyWhat happened
Running batch evaluation against existing traces fails. The runner fetches items via endpoints that are unavailable on v4 events_only deployments:
scope="traces"→client.api.trace.list→GET /api/public/tracesscope="observations"→ legacyobservations_v1→GET /api/public/observationsBoth return:
Root cause
BatchEvaluationRunner._fetch_batch_with_retry(langfuse/batch_evaluation.py, line 1139 in v4.14.4) still calls the v3 read APIs. Per the v3→v4 migration guide, reads should go throughGET /api/public/v2/observations(client-side grouping bytraceId, root observation for trace-level io).Expected
batch_evaluationworks onevents_onlydeployments, e.g. by fetching via the v2 observations API.Workaround
We currently run our own thin loop: fetch v2 observations grouped by
traceId→ evaluate →create_score(score-create ingestion works fine on v4).Happy to contribute a PR if the v2-based approach sounds right — glad to align on the design first.