Proposal: a conformance harness for real inference behavior #454

Problem

A typed SDK can be correct against the API schema and still be unreliable for real inference work. Provider behavior changes at the edges: streaming may end differently, tool calls may be partial, logprobs have nested shapes, context limits may truncate or reject a request, and usage fields can be missing when a request fails halfway through.

Those differences are hard to see from unit tests that validate one response at a time. They show up later as broken agent loops, incorrect accounting, or a client that silently interprets a response differently after an API rollout.

Proposal

Add an opt-in provider conformance harness to the new SDK. It would run a small canonical request corpus and save a versioned behavior report for each model or endpoint.

The report should cover:

• normal and streamed chat responses
• tool-call start, delta, completion, and interruption behavior
• logprob nesting and token alignment
• context-length truncation versus error behavior
• finish reasons and usage accounting
• retryable failures and partial responses
• request and response schema versions

The output should be a diffable artifact, not a dashboard. A change in a model or API rollout should be visible as a contract change with the exact request, response shape, and expected compatibility impact. Raw prompts and completions should be redacted or represented by hashes unless the user explicitly opts in.

Suggested first slice

1. Define a small provider-behavior report schema.
2. Add deterministic mock streams for the edge cases above.
3. Add a conformance runner that works without an API key against those fixtures.
4. Add an optional live mode that records sanitized observations from a real endpoint.
5. Make report diffs usable in CI so SDK changes and API rollouts can be checked before release.

This should stay separate from the request and response models. It is a diagnostic and compatibility layer around them.

Why this matters

Together is operating across many models and inference modes. A schema tells users what a response is supposed to look like. A behavior report tells them what actually changed. That distinction matters for production agents, long-context requests, streaming consumers, and eval pipelines.

I have been working on typed chat-completion and logprob contracts in the older client, and I have built serving and evaluation systems where partial failures and latency behavior needed to be inspected after the run. I would be interested in taking the SDK-side report format and fixture runner through a focused implementation if this is useful for the new client.

Is this the kind of compatibility problem the Together team would want represented in the new SDK?