Skip to content

Pino: in-process LogtailStream for serverless handlers - #168

Merged
PetrHeinz merged 10 commits into
masterfrom
claude/pino-stream
Oct 3, 2026
Merged

PetrHeinz merged 10 commits into
masterfrom
claude/pino-stream

Conversation

@PetrHeinz

@PetrHeinz PetrHeinz commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

A Pino transport runs in Pino's worker thread, so a serverless handler cannot wait for its logs. pino.flush() only moves the main thread's buffer into the worker, and the worker's batch and HTTP request freeze with the function (#112).

@logtail/pino now also exports LogtailStream, an in-process Pino destination that logs through a @logtail/node client the application owns:

const logtail = new Logtail("$SOURCE_TOKEN");
const logger = pino(new LogtailStream(logtail));
// at the end of the handler
await logtail.flush(); // or logger.flush(callback)
  • It uses the same mapping as the worker transport (levels, custom level names, messageKey, message_field, Pino's time), in a helper shared by both. Pino 8.21+ shares its level names and message key with the stream through its configuration message; the customLevels option covers older Pino.
  • Lines are handled synchronously in _write, so a flush right after a log call covers that log.
  • logger.flush() calls the stream's flush(callback), which flushes the client. Ending the stream flushes it too.
  • context.runtime points at the caller, including for custom-level methods.
  • Lines that are not JSON are skipped, as the worker transport does.
  • A failed send does not become an unhandled rejection; it is counted in logtail.dropped. With throwExceptions there is no caller to throw to, and printing the error would get it forwarded again by replaceConsoleMethods().

Two fixes in the shared mapping apply to the worker transport too:

  • pino({ timestamp: false }) no longer throws; it used to kill the worker silently.
  • dt is the time Pino logged, also for its default epoch milliseconds; before, it was the processing time. A numeric time below 1e11 is read as epoch seconds (pino.stdTimeFunctions.unixTime).

@logtail/node's flush() now also waits for sends already in flight, such as a batch the 1 s timer had already started, so everything logged before the call has been sent when it resolves. If one of those sends fails, it flushes once more, so the logs the batcher put back go out before it resolves. Core keeps the started sends until they settle. Browser and Edge flush as before.

This is patch-level:

  • No options change.
  • The stream gains flush(callback).
  • Node's flush() resolves later only while an earlier send is still running or being resent, which is bounded by timeout and retryCount.

Verified against Better Stack on Node 20, 22 and 26 and Pino 8.16 to 10.3, with a harness that freezes the process the moment the handler resolves, as Lambda does:

  • Stream + await logtail.flush(): 16 of 16 lines delivered before the freeze, also with a 1.5 s handler whose timer batch is still in flight. The worker transport delivers 2 of 16, with or without logger.flush().
  • Stream + logger.flush(): 16 of 16.
  • throwExceptions with a revoked token: the process keeps running; before, it died with an unhandled rejection.
  • A send in flight that fails, followed by an immediate exit: 5 of 5 delivered through the resend.

Each fix commit is preceded by a tests-only commit that is expected to fail CI.

Docs: the Pino page needs a "Serverless" section showing the stream and await logtail.flush().

🤖 Generated with Claude Code

PetrHeinz and others added 2 commits September 30, 2026 17:47
Pins a Pino destination that logs through a Logtail client the application
owns, with the same level, custom level and messageKey mapping as the
transport, synchronous enough for `await logtail.flush()` to cover it, and
flushing when the stream ends.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A Writable Pino destination logging through a @logtail/node client the
application owns, so `await logtail.flush()` covers everything logged
before a serverless handler returns. The line mapping (levels, custom
level names, messageKey, message_field, time) moves to a shared helper
used by both the worker transport and the stream; the stream picks up
Pino's configuration from the message Pino emits on its destination.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@PetrHeinz
PetrHeinz marked this pull request as ready for review September 30, 2026 15:59
PetrHeinz and others added 2 commits September 30, 2026 18:44
Pins that a custom level method such as logger.notice() still gets
context.runtime pointing at its caller, that a logger with
`timestamp: false` delivers its logs through both the stream and the
worker transport, and that dt is the time Pino wrote, for Pino's default
epoch milliseconds as well as ISO strings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pino's level methods, custom ones included, are all its function named
LOG, so the stack hint now matches that frame too and a call such as
logger.notice() gets context.runtime pointing at its caller.

The time mapping read obj.time.length, which threw when Pino writes no
time (`timestamp: false`): through the stream the error reached the
caller of the log method, and it killed the transport worker silently.
It also skipped Pino's default epoch milliseconds, so dt was the time
the log was processed. Any time Pino wrote now goes through new Date(),
and dt is set when that is a valid date, for the stream and the worker
transport alike.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PetrHeinz and others added 5 commits October 1, 2026 16:38
…gger.flush, sends in flight

- A failed send through LogtailStream with throwExceptions must not be an unhandled rejection
- Epoch seconds (pino.stdTimeFunctions.unixTime) become dt, in the stream and the worker transport
- A line that is not JSON is skipped and the stream keeps logging
- logger.flush(callback) flushes the Logtail client behind the stream
- Node flush() waits for a send the batcher already started; core tracks sends until they settle

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…poch seconds

- LogtailStream catches the rejection of a failed send under throwExceptions instead of leaving it
  unhandled, which killed the process (every Lambda invocation failed with Runtime.UnhandledPromiseRejection).
  The failure stays counted in logtail.dropped; it is not printed with console.error, because
  replaceConsoleMethods() on the same client would log it once more.
- A line that is not JSON is skipped instead of throwing in _write, which left the stream wedged with every
  later log buffered.
- LogtailStream.flush(callback) flushes the Logtail client, so logger.flush(callback) waits for the logs.
- A numeric time below 1e11 is epoch seconds (pino.stdTimeFunctions.unixTime) and multiplied by 1000,
  in the stream and the worker transport; such values as milliseconds would be before March 1973.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Core keeps the sends the batcher started in a set until they settle (both outcomes handled, so no
unhandled rejection). Node's flush() snapshots them, flushes, then waits for the snapshot with failures
ignored, as the batcher retries or drops a failed batch itself. A serverless handler that awaits
logtail.flush() is therefore not frozen while a batch sent by batchInterval or batchSize is in flight.
Browser and Edge flush() are unchanged; tools is untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A send already in flight when flush() is called fails; its logs go back to the batcher for a retry on the
batch timer, and flush() must not resolve before they were sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a send that was already in flight fails, the batcher puts its logs back into the buffer for a retry on the
batch timer, so flush() resolved while they were still unsent and a process exiting right after it lost them.
flush() now notes which of the sends it waited for failed and, if any did, flushes once more: that sends the
returned logs right away, and a failure of that send goes through the batcher's usual retries before flush()
resolves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	packages/core/src/base.test.ts
#	packages/core/src/base.ts
@PetrHeinz
PetrHeinz merged commit fcbfb5b into master Oct 3, 2026
6 checks passed
@PetrHeinz
PetrHeinz deleted the claude/pino-stream branch October 3, 2026 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant