Nightly build

Automated nightly build from main (0a58e63). These artifacts are unstable and intended for testing only.

v1.5.0

⚡ Highlights

🧮 Transformation rules work on array columns. Now each element is transformed, with an optional array_options to control the output
🔐 New fpe_ff1 transformer: format-preserving encryption for phone numbers, emails and codes, uniqueness preserved
🔎 New lookup_choice transformer: replaces a value with one read from a live table column
🔑 Kafka SASL authentication (plain, scram-sha-256, scram-sha-512) on source and target
⚡ Parallel index restore in pg-to-pg schema snapshots via index_restore_workers, and transient pg_restore failures are retried instead of aborting the snapshot
🧩 Transformers accept more column types: numeric in the greenmask number transformers, user-defined enums in greenmask_choice, any type in template

What's Changed

New features

• Add an array transformer that maps over array elements by @kvch in #1182
• Apply transformation rules to array columns by @kvch in #1184
• Add fpe_ff1 format-preserving transformer for emails, phone numbers by @kvch in #1165
• Add lookup_choice transformer by @kvch in #1148
• Add sasl to kafka by @RemiDesgrange in #1169
• Restore indexes in parallel during pg2pg schema snapshot by @kvch in #1144
• Retry index restore on transient errors by @kvch in #1139
• Support enum columns in greenmask_choice by @kvch in #1192
• Support numeric columns in transformers by @kvch in #1140
• Allow any column type in template transformer by @kvch in #1185
• Add context to transformer validation errors by @kvch in #1176

Bug fixes

• Stop retrying data exceptions and name the column by @kvch in #1175
• Bind the env vars the configuration reads by @kvch in #1145
• Encode snapshot values with pgx text codecs in templates by @kvch in #1188
• Parse range literals for binary COPY in bulk ingest by @kvch in #1187
• Decode array elements with a catalog aware type map by @kvch in #1189
• Warn and fail early on array column rule surprises by @kvch in #1190
• Resolve enum types and return choice labels as strings by @kvch in #1191

Dependency updates

• Bump github.com/moby/go-archive from 0.2.0 to 0.3.0 by @dependabot[bot] in #1171
• Bump go.opentelemetry.io/contrib/instrumentation/runtime from 0.70.0 to 0.71.0 by @dependabot[bot] in #1154
• Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc from 1.45.0 to 1.46.0 by @dependabot[bot] in #1155
• Bump clickhouse/clickhouse-server from 26.7.5.10-alpine to 26.8.2.7-alpine by @dependabot[bot] in #1150

Details

Transformation rules on array columns

A rule on an array column such as varchar(500)[] was rejected at validation time with does not support pg data type: _varchar with OID: 1015. Because the rejection happened for the whole rule set, a single array column blocked transformation of every other column in the table. A rule on a one-dimensional array column now applies the transformer to each element.

transformations:
  table_transformers:
    - schema: public
      table: users
      column_transformers:
        emails: # varchar(500)[]
          name: email
          parameters:
            replacement_domain: "@example.com.invalid"

pgstream decodes the array before it calls the transformer, so each element arrives as the value of its element type: an int4[] element is an integer, not the text "1", and a NULL element is distinguishable from the string "NULL". A transformer accepts an array column if it accepts the element type, which extends to extension types (citext[]) and to arrays of enums. The array is re-encoded in the same shape it arrived in, so Kafka, webhook and search payloads for array columns are unchanged.

The optional array_options block controls how the output array is built:

column_transformers:
  emails:
    name: email
    array_options:
      generator: random # map (default) or random
      min_count: 0
      max_count: 12

map applies the transformer to each element in order and keeps the length. random emits between min_count and max_count elements, each the transform of a source element drawn at random with replacement, so it never invents a value that was not in the row. min_count and max_count are required with random, rejected with map, and max_count is capped at 10000. Array columns need a source PostgreSQL connection, since the element type is read from the catalog; the connection-less parser rejects array_options.

The uniqueness check reads the classification of the whole array transform: with map the array keeps the transformer's classification, so fpe_ff1 on text[] stays preserved; with random the array is always lossy, since min_count: 0 lets any row produce an empty array. A NULL array stays NULL, a NULL element stays NULL, and the first failing element fails the whole column under the configured on_error policy. Multi-dimensional arrays are not supported and are rejected at startup when declared as such.

Three things the validator now flags: literal_string and pg_anonymizer write into every element on an array column, which is a change for a configuration written before array support; pg_anonymizer sends one query per element, so a wide array multiplies the source load; and a column declared with more than one dimension is rejected up front instead of failing row by row. The first two are warnings in pgstream validate rules and in the run log.

fpe_ff1: format-preserving encryption

The output has the same length and the same characters as the input. An encrypted phone number is still a phone number, an encrypted product code still fits its varchar(n) column, and FF1 (NIST SP 800-38G) maps each input to one different output, so the transformer is classified preserved and can be used on a unique-indexed column. It is the first uniqueness-preserving transformer whose output keeps the original format.

transformations:
  table_transformers:
    - schema: public
      table: persons
      column_transformers:
        phone_number:
          name: fpe_ff1
          parameters:
            key_hex: ${FPE_KEY} # 32, 48 or 64 hex chars; openssl rand -hex 32
            associated_data: public.persons.phone_number # FF1 tweak, not secret
            alphabet: digits # digits, letters, alphanumeric, or a set of characters
            passthrough: keep # copy characters outside the alphabet to the output
            keep_prefix: 4 # keep the country and operator code
        email:
          name: fpe_ff1
          parameters:
            key_hex: ${FPE_KEY}
            associated_data: public.persons.email
            alphabet: alphanumeric
            passthrough: keep
            preserve_from: "." # keep the top-level domain

keep_prefix and keep_suffix count only characters in the alphabet, so one rule serves every phone format: +36301234567, +36 30 123 4567 and +36-30-123-4567 all keep their first four digits and their separators. preserve_from keeps the end of the value from the last occurrence of a delimiter: "." keeps the top-level domain so the address stays well-formed, "@" keeps the full domain but then the target contains every source domain. FF1 refuses domains smaller than 10⁶, so a value needs at least 6 digits, or 4 letters or alphanumerics, after prefix, suffix and passthrough characters are removed; shorter values go to the on_error policy rather than reaching the target in clear.

Like encrypted_aes_siv, this is pseudonymization: anyone holding key_hex can decrypt, and equal inputs give equal outputs. Keep the key in a secret store, use a different key per environment, and set associated_data per column so tokens cannot be correlated across columns.

lookup_choice: choose from a live table

greenmask_choice and greenmask_integer force users to regenerate their rules file whenever a lookup table changes. lookup_choice replaces a value with one taken from a column of another table, read from the source database once at pipeline start:

transformations:
  table_transformers:
    - schema: public
      table: addresses
      column_transformers:
        country_id:
          name: lookup_choice
          parameters:
            lookup_table: public.countries # schema-qualified, quoted as written
            lookup_column: id
            generator: deterministic # or random
            ignore_values: [0, -1] # drop placeholder rows from the pool
            # max_values: 100000   # load fails above this, rather than truncating
            # postgres_url: ...    # only needed when the source is not PostgreSQL

The type is taken from the lookup column, and a rule that points a column at an incompatible lookup type is rejected at startup; a narrower integer or float is accepted for a wider column. deterministic picks by hash of the incoming value so rows that shared an original still share a replacement, `ra...

v1.4.2

⚡ Highlights

🧭 Replicated DDL is applied under the schema it was issued in, so unqualified CREATE TABLE no longer lands in public on the target
🕰 timetz columns are supported
🛂 pgstream check no longer fails its CREATEROLE check with database "<role>" does not exist
🔌 Dropping the database from a connection string no longer drops the user with it, on the pg_restore --create path too
Dependency updates

Bug fixes

• Replay DDL with the upstream search path by @kvch in #1131
• Fix create role check when database is empty by @kvch in #1134
• Add support for timetz by @kvch in #1135

Dependency updates

• Bump github.com/bytedance/sonic from 1.15.1 to 1.15.3 by @dependabot[bot] in #1121
• Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc from 1.44.0 to 1.45.0 by @dependabot[bot] in #1123
• Bump go.opentelemetry.io/contrib/instrumentation/runtime from 0.69.0 to 0.70.0 by @dependabot[bot] in #1112
• Bump github.com/tink-crypto/tink-go/v2 from 2.7.0 to 2.8.0 by @dependabot[bot] in #1111
• Bump github.com/elastic/go-elasticsearch/v8 from 8.19.6 to 8.19.7 by @dependabot[bot] in #1113
• Bump github.com/stretchr/testify from 1.11.1 to 1.12.1 by @dependabot[bot] in #1115
• Bump github.com/testcontainers/testcontainers-go/modules/kafka from 0.43.0 to 0.44.0 by @dependabot[bot] in #1117
• Bump clickhouse/clickhouse-server from 26.7.3.19-alpine to 26.7.5.10-alpine by @dependabot[bot] in #1120

Full Changelog: v1.4.1...v1.4.2

v1.4.1

⚡ Highlights

🧬 Replicated bytea values are no longer corrupted on the way to a Postgres target
🧩 Partitioned tables restore with valid indexes and primary keys, so they keep replicating after a snapshot
🔐 Binaries are built with Go 1.26.6, patching stdlib and module CVEs
Dependency updates

What's Changed

Bug fixes

• Decode bytea hex from wal2json before writing to postgres by @siriusfrk in #1090
• Restore ALTER INDEX ... ATTACH PARTITION with indices and constraints by @siriusfrk in #1094

Dependency updates

• Bump Go to 1.26.6 and patch stdlib / module CVEs by @walidelbiir in #1107
• Bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc from 1.44.0 to 1.45.0 by @dependabot[bot] in #1097
• Bump go.opentelemetry.io/otel/exporters/prometheus from 0.66.0 to 0.67.0 by @dependabot[bot] in #1099
• Bump testcontainers-go postgres, opensearch and elasticsearch modules from 0.43.0 to 0.44.0 by @dependabot[bot] in #1098, #1100, #1096
• Bump clickhouse/clickhouse-server from 26.7.2.59-alpine to 26.7.3.19-alpine by @dependabot[bot] in #1095

New Contributors

• @siriusfreak made their first contribution in #1090

Full Changelog: v1.4.0...v1.4.1

v1.4.0

⚡ Highlights

📊 Native Prometheus scrape endpoint
🎯 Granular object type filtering for pg-to-pg schema snapshots and DDL replication
🛡️ Transformation rules are now validated against unique indexes before a run starts, instead of failing mid-load
🔌 Target connection pool limits are configurable
🪝 Webhook delivery retries transient failures and no longer silently drops still-failing events; adds idempotency headers
🐛 Fixed a DDL parsing bug that silently dropped RENAME COLUMN / DROP COLUMN schema changes
🛂 New preflight check for CREATEROLE when a snapshot restores roles
Dependency updates

What's Changed

New features

• Add prometheus endpoint by @pauliyobo in #1008
• Granular object type filtering for pg to pg snapshot and streaming by @tsg in #735
• Add target CREATEROLE preflight check for roles snapshotting by @lghuy05 in #1055
• Make target connection limits configurable by @subotac in #1048
• Add a uniqueness classification to transformers by @kvch in #1082
• Classify the greenmask and neosync transformers for uniqueness by @kvch in #1084, #1085
• Validate transformation rules against unique indexes by @kvch in #1086

Bug fixes

• Reject masking configurations that mask no characters by @kvch in #1083
• Fix retry idempotency for webhook target by @kvch in #1080
• Stop the statement terminator being captured as part of a column name by @kvch in #1078

Documentation

• Document transformer uniqueness and the unique index check by @kvch in #1088
• Document the destructive --reset snapshot behaviour by @kvch in #1059

Dependency updates

• Bump actions/attest from 4.2.0 to 4.2.2 by @dependabot[bot] in #1065
• Bump go.opentelemetry.io/otel/sdk/metric from 1.44.0 to 1.45.0 by @dependabot[bot] in #1064
• Bump github.com/pgvector/pgvector-go/pgx from 0.4.0 to 0.4.1 by @dependabot[bot] in #1066
• Bump docker/login-action from 4.5.2 to 4.6.0 by @dependabot[bot] in #1061
• Bump clickhouse/clickhouse-server, opensearch and opensearch-dashboards build images by @dependabot[bot] in #1060, #1062, #1063

Details

Prometheus scrape endpoint

The setting instrumentation.metrics.prometheus.enabled exposes a /metrics endpoint on the existing health server. Metrics can now be scraped directly, with no collector in between:

instrumentation:
  metrics:
    prometheus:
      enabled: true # Defaults to false
      endpoint: "/metrics" # Defaults to /metrics

The endpoint returns 404 when disabled.

Object type filtering

Both the schema snapshot and DDL replication for pg-to-pg pipelines can now be restricted to a subset of object categories (tables, sequences, types, indexes, functions, views, triggers, etc.) via include_object_types (allowlist) or exclude_object_types (denylist), independently for snapshot and replication. A DDL statement that touches several object types (e.g. CREATE TABLE with a primary key) is only skipped when every object it touches is excluded.

Transformation rules are validated against unique indexes.

Anonymization is lossy by design: masking with type: id keeps a 6-character prefix, so every value sharing that prefix collapses to the same masked value. Applied to a column covered by a unique index, this previously surfaced as duplicate key value violates unique constraint hours into a data load. Every transformer is now classified as preserved (distinct inputs stay distinct), not_guaranteed (random or hashed output, collisions possible) or lossy (collides by construction).

validate rules and pipeline startup now read pg_index for the tables in scope and check the configured rules against unique indexes, primary keys and unique constraints. With a Postgres target, a lossy transformer on a covered column fails at parse time with transformation rules break a unique index; with other targets (Kafka, Elasticsearch/OpenSearch, webhooks) the same finding is a warning, since there is no unique index to violate. A not_guaranteed transformer always warns. Set allow_uniqueness_loss: true on a column rule to keep a lossy transformer anyway. A custom mask configured to mask zero characters (mask_begin equal to mask_end, or unmask_begin: 0) is now rejected at construction rather than silently passing values through unmasked. Use noop to pass a column through on purpose.

Configure target side connection count

target.postgres.max_connections (PGSTREAM_POSTGRES_WRITER_MAX_CONNECTIONS) now controls the writer pool size, defaulting to 50 and honoring pool_max_conns from the target URL when set explicitly. Previously this setting also sized the schema observer's pool, so a target capped at 90 connections to stay under a 100-connection server limit could still open up to 180. The schema observer is now sized separately, at min(writer pool, 16). So the process now opens at most the configured writer limit plus 16.

Webhook delivery

A failed webhook POST is now retried with configurable backoff (network errors, 429, 5xx); other non-2xx responses are treated as permanent and not retried. A delivery that is still failing after retries is logged and dropped so one broken subscriber can't block delivery to everyone else, and at-least-once semantics apply on restart. New X-Pgstream-LSN / Idempotency-Key headers let subscribers dedupe redelivered events (omitted for snapshot events, which share a zero LSN). DISABLE_RETRIES now actually disables retries.

New Contributors

• @pauliyobo made their first contribution in #1008
• @subotac made their first contribution in #1048

Full Changelog: v1.3.1...v1.4.0

v1.3.1

⚡ Highlights

📌 Data snapshots read the columns the schema snapshot captured, so a column added to the source mid-snapshot no longer breaks the load
⚠️ Schema drift during a snapshot is now reported: a warning when it happens during the dump, an error when a table's captured columns are gone
🛂 New preflight check for CREATEDB on the target when the snapshot creates the target database
🔐 Documented how DDL replication can carry privileges from a less trusted source to the target
Dependency updates

What's Changed

New features

• Add target CREATEDB preflight check for snapshot DB creation by @lghuy05 in #996

Bug fixes

• Pin the data snapshot to the columns the schema snapshot saw by @kvch in #1058
• Add catalog helpers for reading a table's column list by @kvch in #1057

Documentation

• Document potential privileges escalation between source and target via DDL replication by @tsg in #1052

Dependency updates

• Bump docker/login-action from 4 to 4.5.2 by @dependabot[bot] in #1054
• Bump actions/attest from 4 to 4.2.0 by @dependabot[bot] in #1053

Full Changelog: v1.3.0...v1.3.1

v1.3.0

Changelog

• 474ff1c Added tests and docs for snapshot+stream from a read replica (#1049)
• c06f9bd Fix encoding of empty strings as null bytes (#1051)

What's Changed

• Added tests and docs for snapshot+stream from a read replica by @tsg in #1049
• Fix encoding of empty strings as null bytes by @tsg in #1051

Full Changelog: v1.2.5...v1.3.0

v1.2.5

⚡ Highlights

🎲 Template and JSON transformers no longer corrupt their random generator under multi-worker snapshots
📉 New metrics and totals for what ignore_send_errors silently discards, plus a startup warning when suppression is on
🧾 Snapshot failures are logged as errors instead of rendering as {}
🔁 Snapshot request status updates are retried, and report data loss when they still cannot be persisted
🔍 A failed restore now carries the output that explains it, with credentials redacted

What's Changed

Bug fixes

• Pool the templates that share greenmask's funcmap state by @kvch in #1045
• Add a counter for batches dropped by ignore_send_errors by @kvch in #1042
• Log snapshot failures as errors instead of rendering them as {} by @kvch in #1039
• Retry snapshot recorder requests, emit errors by @kvch in #1041
• Add more details about failed statements by @kvch in #1040

CI / build

• Catch both outcomes of the enum type creation race in integration tests by @kvch in #1044
• Update golangci-lint in Makefile to match CI version by @kvch in #1046

Upgrade notes

No configuration changes are required.

Contributors. make lint now installs golangci-lint v2.10.1 to match CI, which reports findings the previously pinned v2.5.0 did not.

Full Changelog: v1.2.4...v1.2.5

v1.2.4

⚡ Highlights

🔑 Replicated DELETEs on tables with a user-defined enum primary key no longer fail
🧬 Enum arrays and domains over enums are now handled in delete identities, not just plain enums
🛟 A failed query can no longer take its whole batch down with it, including writes to unrelated tables

What's Changed

Bug fixes

• Fix replicated DELETEs on user-defined enum PKs by @kvch in #1038

Upgrade notes

No configuration changes are required, and the SQL emitted for tables without enum identity columns is unchanged.

Full Changelog: v1.2.3...v1.2.4

v1.2.3

⚡ Highlights

🎨 Bulk ingest no longer fails on tables with user-defined enum columns
🔑 Kafka primary_key partitioning no longer collides distinct rows onto one message key
🩺 Two preflight checks that reported false failures on correctly-configured sources (wal2json, replica identity) are fixed
🔍 Preflight can now run source-only checks (--source) and verifies major PostgreSQL version compatibility
Dependency updates

What's Changed

New features

• [Preflight] Option to run only source side checks by @kvch in #1025
• [Preflight] Check major PostgreSQL version by @kvch in #1026

Bug fixes

• Copy user defined enum columns by @kvch in #1035
• Escape keys when using primary key partitioning by @kvch in #1036
• Fix two false-result preflight checks (wal2json, replica_identity) by @kvch in #1034
• Update PostgreSQL version check for source size support by @kvch in #1027

Dependency updates

• Bump actions/github-script from 7 to 9 by @dependabot[bot] in #1030
• Bump actions/download-artifact from 7 to 8 by @dependabot[bot] in #1029
• Bump clickhouse/clickhouse-server from 26.6.1.1193-alpine to 26.7.1.1315-alpine in /build/docker by @dependabot[bot] in #1028

Full Changelog: v1.2.2...v1.2.3