Skip to content

Kafka connect runtime 1.12.0 Azure Blob writes silently hang due to netty-tcnative/netty-handler #18395

Open

Description

@Ycallaer

Apache Iceberg version

1.12.0 (latest release)

Query engine

Kafka Connect

Please describe the bug 🐞

build.gradle.txt

Hi,
We recently tried upgrading our kafka connect version from 1.11.0 to 1.12.0 but ran into an issue.
Our setup is as follows:

  • kafka connect CFK 8.3.1
  • Catalog : Databricks
  • Behind the catalog is an azure blobstorage account

When we moved to 1.12.0 ( no other changes were done in connect) we saw the connector running but no data was being forwarded.
When we put connect in debug mode we could see the following failure

java.lang.NoSuchMethodError: 'java.lang.String io.netty.internal.tcnative.SSL.getGroupName(long)'
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.handshakeFinished(ReferenceCountedOpenSslEngine.java:2658)
	at io.netty.handler.ssl.ExtendedOpenSslSession.handshakeFinished(ExtendedOpenSslSession.java:251)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.handshake(ReferenceCountedOpenSslEngine.java:2025)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:889)
	...

I did some analysis with AI on this ( as I am not super familiar with the code base) and we saw the following:

On Iceberg 1.11.0, the iceberg-kafka-connect-runtime distribution resolved these versions together:

  • netty-handler:4.2.13.Final
  • netty-tcnative-classes:2.0.74.Final / netty-tcnative-boringssl-static:2.0.74.Final

These are mutually compatible, and Azure Blob Storage writes (via iceberg.catalog.vended-credentials-enabled=true, which routes file I/O through azure-core-http-netty/reactor-netty) worked correctly.

In 1.12.0:
gradle/libs.versions.toml bumped azuresdk-bom from 1.3.6 to 1.3.8. This transitively pulls a newer netty-handler:4.2.18.Final (via azure-core-http-netty/reactor-netty), but netty-tcnative-classes/netty-tcnative-boringssl-static only moved to 2.0.78.Final — three releases behind what netty-handler:4.2.18.Final actually needs.
netty-handler:4.2.18.Final's ReferenceCountedOpenSslEngine$DefaultOpenSslSession.handshakeFinished() calls:
io.netty.internal.tcnative.SSL.getGroupName(long)
This method was only added upstream in netty-tcnative 2.0.81.Final (netty/netty-tcnative#991 (netty/netty-tcnative#991), merged 2026-07-07). It does not exist in 2.0.74.Final through 2.0.80.Final.

Impact
Any Kafka Connect sink task writing to Azure Blob Storage with iceberg.catalog.vended-credentials-enabled=true hangs silently and permanently on its first TLS handshake to blob storage:
java.lang.NoSuchMethodError: 'java.lang.String io.netty.internal.tcnative.SSL.getGroupName(long)'
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.handshakeFinished(ReferenceCountedOpenSslEngine.java:2658)
at io.netty.handler.ssl.ExtendedOpenSslSession.handshakeFinished(ExtendedOpenSslSession.java:251)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.handshake(ReferenceCountedOpenSslEngine.java:2025)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:889)
...

The critical part: the exception happens on a reactor-netty I/O thread, not the Kafka Connect task thread, and is caught by reactor.core.Exceptions.throwIfFatal, which logs it only at WARN:
throwIfFatal detected a jvm fatal exception, which is thrown and logged below:
java.lang.NoSuchMethodError: 'java.lang.String io.netty.internal.tcnative.SSL.getGroupName(long)'
It is never propagated back to the sink task. The task's own thread is left waiting on a future that will never complete. Kafka Connect shows the connector/task as RUNNING with no ERROR-level log at all. The consumer keeps polling and advancing offsets, the sink writer successfully builds Parquet files in memory, but the coordinator's commit never gets a DataWritten event, so every commit cycle logs:
Coordinator ... completed commit , committed to 0 table(s), valid-through null
indefinitely. From the outside, data appears to stop flowing into the table with zero diagnostic signal — only visible at DEBUG level, where the affected task's thread simply stops logging mid-stream while other threads/connectors continue.

Our fix
Forcing netty-tcnative-classes and netty-tcnative-boringssl-static to 2.0.81.Final (the first version with getGroupName) in the kafka-connect-runtime module's dependency resolution resolves the NoSuchMethodError. We also had to explicitly add a runtimeOnly dependency on the classified netty-tcnative-boringssl-static:...:linux-x86_64 artifact, since the default (classifier-less) artifact Gradle resolves transitively contains no native .so at all — it's a metadata-only jar.

Suggested fix upstream
In kafka-connect/build.gradle, bump netty-tcnative-classes/netty-tcnative-boringssl-static to at least 2.0.81.Final to match the netty-handler version actually being shipped, and ensure the platform-classified native artifact is included in the runtime distribution.
I have added the build.gradle.txt ( remove txt extension) which worked for us

Willingness to contribute

  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time

Activity

  1. developer-rpai commented on Oct 6, 2026

    @developer-rpai
    Contributor

    Looking into this — will follow up with a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions