"Cancelled" streams causing gRPC status code 13 in Envoy proxy

[bwmaas]

I see an info level [n] stream cancelled in my xDS server logs (where the [n] is the stream id) immediately followed by a warning level StreamEndpoints gRPC config stream closed: 13 in my Envoy logs. I'm running v0.1.31 of java-control-plane and 1.18.6 of Envoy. Envoy appears to be getting the data successfully but logs are getting filled on both sides of the connection. It "feels" like something is abruptly ending the gRPC connection.

My reading of the code suggests the [n] stream cancelled message is a red herring. It appears this will be outputted whenever a stream is cancelled (the DiscoveryRequestStreamObserver.onCancelled method) and that the DiscoveryRequestStreamObserver.cancel method is called whenever a stream closed, whether with an error or not. The cancel method appears to only cancel watchers though.

I'm trying to collect more information on this issue. I'll try getting a debugger on the xDS server tomorrow.

[ZIK-1337]

hi, any update? I have the same problem in the last version

[liquidnya]

I was having the same issue and figured out what has caused it for me and how to fix it.

The cause:
TL;DR The HTTP/2 connection was closed by the control plane (xDS server).

In my case envoy was configured to send PING packets every 30 seconds:

http2_protocol_options:
  connection_keepalive:
    interval: 30s
    timeout: 5s

See config.core.v3.KeepaliveSettings, which is configured in config.core.v3.Http2ProtocolOptions which is configured in config.cluster.v3.Cluster when configuring the control plane (xDS server) in the bootstrap configuration of envoy.

And the gRPC server of the control plane (xDS server) was configured to only allow one PING packet every 5 minutes.
Envoy sent a ping packet every 30 seconds, which is not greater than 5 minutes, which resulted into ping strikes.
After 3 ping strikes the control plane (xDS server) closed the HTTP/2 connection which also results in closing all streams on that connection.
This resulted in log messages every 90 seconds and also unnecessary reconnects.

The Java class that caused this behavior is io.grpc.internal.KeepAliveEnforcer.

The fix:

Look at the Keepalive guide in gRPC: https://grpc.io/docs/guides/keepalive/
Choose values for:

• PERMIT_KEEPALIVE_TIME on the xDS server.
• KEEPALIVE_TIME on the xDS client (control plane cluster in the envoy bootstrap config) and xDS server.
• KEEPALIVE_TIMEOUT on the xDS client (control plane cluster in the envoy bootstrap config) and xDS server.

Make sure that the PERMIT_KEEPALIVE_TIME on the xDS server is smaller than KEEPALIVE_TIME on the xDS client!
Now configure those values for the xDS client (control plane cluster in the envoy bootstrap config) and xDS server.

Example:

Note that this example might not fit your use-case.

I was setting the following settings for your control plane cluster in the envoy bootstrap config:

http2_protocol_options:
  connection_keepalive:
    interval: 30s # KEEPALIVE_TIME
    timeout: 5s # KEEPALIVE_TIMEOUT

When building the gRPC server in Java I used the io.grpc.netty.NettyServerBuilder and called the following methods:

NettyServerBuilder nettyServerBuilder = NettyServerBuilder.forPort(18000);
// PERMIT_KEEPALIVE_TIME
nettyServerBuilder.permitKeepAliveTime(10, TimeUnit.SECONDS);
// KEEPALIVE_TIME
nettyServerBuilder.keepAliveTime(30, TimeUnit.SECONDS);
// KEEPALIVE_TIMEOUT
nettyServerBuilder.keepAliveTimeout(20, TimeUnit.SECONDS);

Note that different libraries/frameworks use a different way of configuring these. E.g. the quarkus.grpc.server.netty.keep-alive-time and quarkus.grpc.server.netty.permit-keep-alive-time properties in Quarkus.

Just make sure that the PERMIT_KEEPALIVE_TIME on the server is less than the KEEPALIVE_TIME on the client.

[bwmaas]

That seems like a reasonable fix! I totally forgot about this issue.

[ZIK-1337]

@liquidnya but if connection_keepalive.interval is not set in the envoy config, then this is not the problem?

[ZIK-1337]

@liquidnya but if connection_keepalive.interval is not set in the envoy config, then this is not the problem?

@liquidnya what do u think?