Generate audit and saga load against a ServiceControl.Audit instance

[johnsimons]

The harness only drives error ingestion today. This adds the audit side: a direct audit-queue writer, an Aspire audit instance running on EF persistence, and audit series on the Grafana dashboard beside the error ones.

Audit ingestion lives in the ServiceControl.Audit instance rather than the primary, so the load targets the audit container's own queue.

Direct audit-queue writer

New endpoints alongside the existing bypass writer:

• POST /api/audit-bypass/start with { "rate": 100, "parallelism": 4, "sagaSnapshotRatio": 0.2, "durationSeconds": 60 }
• POST /api/audit-bypass/stop
• GET /api/audit-bypass/status

Most claims emit one processed-message envelope carrying the headers a real endpoint would have stamped: NServiceBus.ProcessingEndpoint, the host ids, and TimeSent, ProcessingStarted and ProcessingEnded spread so delivery, processing and critical times are not zero. They rotate across a pool of simulated endpoints, message types and conversations.

The writer shares the error writer's pacing. BypassRun is split into a LoadRun base holding the clock, the shared claim budget and AchievedRate, with BypassRun adding the scenario. DirectErrorQueueWriter behaviour is unchanged.

Saga episodes

A share of claims emit a whole saga episode instead: the three messages that drove one saga through New, Updated and Completed, each followed by the snapshot it produced. One worker writes the episode so the phases reach the queue in order.

The links between the pieces are strict, and getting any of them wrong fails quietly:

• Each driving message carries NServiceBus.InvokedSagas (SagaType:SagaId) and ServiceControl.SagaStateChange (SagaId:New|Updated|Completed). InvokedSagasParser is the only path by which an audited message is attached to a saga, and it runs at query time inside ToMessagesView, so a malformed value throws after the rows are already stored.
• Each snapshot's Initiator.InitiatingMessageId is the id of the message directly before it, so drill-through from the saga view resolves to a real audited message.
• The snapshot body uses exact CLR casing, and its EnclosedMessageTypes header is the bare unqualified type name. AuditPersister compares that header with ==, so an assembly-qualified value is stored as an ordinary audit row with no error and no saga history.

sagaSnapshotRatio is snapshots per processed message. The default 0.2 emits two snapshots for every ten processed messages, and 0 disables episodes.

AppHost

--audit-instances now gives each audit instance EF persistence under the audit settings namespace (SERVICECONTROL_AUDIT_PERSISTENCETYPE, SERVICECONTROL_AUDIT_DATABASE_CONNECTIONSTRING), OTLP export, audit-namespaced ingestion tuning, and ServiceControlQueueAddress so its custom checks and failed audit imports reach the primary. No body storage is configured, because the audit EF persisters store bodies inline.

The first audit instance runs SetupAndRun and owns the migrations. Later instances run and wait for it, so several instances on one database do not race EF migrations. Partition provisioning continues from the retention sweep.

--audit-instances stays defaulted to 0. Asking for audit instances on SQL persistence without --tag prints a warning, because the audit EF persisters are newer than the latest release and the default particular/servicecontrol-audit:latest image does not contain them.

Dashboard

Ingestion panels carry audit series beside the error ones, labelled error: and audit:, using the existing smoothing idiom. Audit message categories are audit-message, saga-update and control-message, so the audit targets filter on those rather than failed-message.

Some panels deliberately differ. The audit instance emits no sc.audit.ingestion.storage_duration_seconds, so the storage write and storage share panels stay error only and say so in their descriptions. The audit heatmap is its own panel, since two bucket distributions on one heatmap are unreadable. Batch outcome share puts audit in a separate stacking group, because each share sums to 1 on its own.

Verification

A harness compiled against the real SagaAuditMessagesSerializationContext, SagaSnapshotFactory and InvokedSagasParser asserts 63 properties of the generated messages: bodies deserialize, the saga-link headers parse to the right status, snapshot and message agree on saga id and type, initiating ids resolve, phases order with increasing finish times, times come back Kind=Utc, and the ratio arithmetic produces the requested mix.

A LearningTransport run wrote 548 processed messages and 108 snapshots covering 36 sagas, spread across all six simulated endpoints and all three saga types, with every message id distinct and the headers intact on the wire.

Known limitation

Running audit load on SQL Server or PostgreSQL needs an audit image containing the EF audit persisters. Until the next release that means --tag with a prerelease, or a locally built image.