Happy eyeballs implementation times out prematurely

Version

v20.12.2

Platform

Linux 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

In an IPv4-only environment:

NODE_DEBUG=net node -e 'fetch ("http://ds-test.r-1.ch:8080/");'

autoSelectFamily : If set to true, it enables a family autodetection algorithm that loosely implements section 5 of RFC 8305. The all option passed to lookup is set to true and the sockets attempts to connect to all obtained IPv6 and IPv4 addresses, in sequence, until a connection is established. The first returned AAAA address is tried first, then the first returned A address, then the second returned AAAA address and so on. Each connection attempt (but the last one) is given the amount of time specified by the autoSelectFamilyAttemptTimeout option before timing out and trying the next address. Ignored if the family option is not 0 or if localAddress is set. Connection errors are not emitted if at least one connection succeeds. If all connections attempts fails, a single AggregateError with all failed attempts is emitted. Default: net.getDefaultAutoSelectFamily()

autoSelectFamily defaults on in nodejs v20.

When trying to establish a connection, this fails to account for a host that has no IPv6 connectivity attempting to connect to a dual-stack host with both A and AAAA records. The DNS server returns responses in the order A, AAAA. Node tries to connect to the A address with only 250ms timeout, insufficient for many real-world cases (cellular/satellite links, poorly connected ISPs, far away servers, packet loss, etc). This times out, so node proceed to the last candidate which is supposed to have a longer timeout, however the last candidate is an AAAA address and the host has no IPv6 connectivity so it immediately fails, causing the overall connection to fail.

How often does it reproduce? Is there a required condition?

Reproduced 100% under the correct test conditions.

What is the expected behavior? Why is that the expected behavior?

Connection should be established successfully.

What do you see instead?

Connection times out with ETIMEDOUT AggregateError.

Additional information

# NODE_DEBUG=net node -e 'fetch ("https://ds.sydney.test-ipv6.com/ip/?callback=?&testdomain=test-ipv6.com&testname=test_ds");'
NET 639: pipe false undefined
NET 639: connect: find host ds.sydney.test-ipv6.com
NET 639: connect: dns options { family: undefined, hints: 32 }
NET 639: connect: autodetecting
NET 639: _read - n 16384 isConnecting? true hasHandle? true
NET 639: _read wait for connection
NET 639: connect/multiple: will try the following addresses [
  { address: '45.79.238.217', family: 4 },
  { address: '2400:8907::f03c:94ff:fed0:3fc3', family: 6 }
]
NET 639: connect/multiple: attempting to connect to 45.79.238.217:443 (addressType: 4)
NET 639: connect/multiple: setting the attempt timeout to 250 ms
NET 639: connect/multiple: connection to 45.79.238.217:443 timed out
NET 639: connect/multiple: attempting to connect to 2400:8907::f03c:94ff:fed0:3fc3:443 (addressType: 6)
NET 639: destroy
NET 639: close
NET 639: close handle
node:internal/deps/undici/undici:12618
    Error.captureStackTrace(err, this);
          ^

TypeError: fetch failed
    at node:internal/deps/undici/undici:12618:11
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: AggregateError [ETIMEDOUT]:
      at internalConnectMultiple (node:net:1116:18)
      at internalConnectMultiple (node:net:1184:5)
      at Timeout.internalConnectMultipleTimeout (node:net:1710:5)
      at listOnTimeout (node:internal/timers:575:11)
      at process.processTimers (node:internal/timers:514:7) {
    code: 'ETIMEDOUT',
    [errors]: [
      Error: connect ETIMEDOUT 45.79.238.217:443
          at createConnectionError (node:net:1646:14)
          at Timeout.internalConnectMultipleTimeout (node:net:1705:38)
          at listOnTimeout (node:internal/timers:575:11)
          at process.processTimers (node:internal/timers:514:7) {
        errno: -110,
        code: 'ETIMEDOUT',
        syscall: 'connect',
        address: '45.79.238.217',
        port: 443
      },
      Error: connect ENETUNREACH 2400:8907::f03c:94ff:fed0:3fc3:443 - Local (:::0)
          at internalConnectMultiple (node:net:1180:16)
          at Timeout.internalConnectMultipleTimeout (node:net:1710:5)
          at listOnTimeout (node:internal/timers:575:11)
          at process.processTimers (node:internal/timers:514:7) {
        errno: -101,
        code: 'ENETUNREACH',
        syscall: 'connect',
        address: '2400:8907::f03c:94ff:fed0:3fc3',
        port: 443
      }
    ]
  }
}

I'm unable to reproduce:

$  NODE_DEBUG=net node -e 'fetch ("https://ds.sydney.test-ipv6.com/ip/?callback=?&testdomain=test-ipv6.com&testname=test_ds");'
NET 86651: pipe false undefined
NET 86651: connect: find host ds.sydney.test-ipv6.com
NET 86651: connect: dns options { family: undefined, hints: 32 }
NET 86651: connect: autodetecting
NET 86651: _read - n 16384 isConnecting? true hasHandle? true
NET 86651: _read wait for connection
NET 86651: connect/multiple: will try the following addresses [
  { address: '45.79.238.217', family: 4 },
  { address: '2400:8907::f03c:94ff:fed0:3fc3', family: 6 }
]
NET 86651: connect/multiple: attempting to connect to 45.79.238.217:443 (addressType: 4)
NET 86651: connect/multiple: setting the attempt timeout to 250 ms
NET 86651: connect/multiple: connection attempt to 45.79.238.217:443 completed with status 0
NET 86651: afterConnect
NET 86651: _read - n 16384 isConnecting? false hasHandle? true
NET 86651: Socket._handle.readStart
NET 86651: _read - n 16384 isConnecting? false hasHandle? true

Note

I only have IPv4

Try a server further away so you can't complete the IPv4 connection in 250ms. You can also try saturating your connection with a speed test to artificially increase your latency.

Some possible alternative servers to test with: https://test-ipv6.com/mirrors.html.en_US

I created a test server that should reproduce this reliably by artificially delaying IPv4 connections.

Try the following:

NODE_DEBUG=net node -e 'fetch ("http://ds-test.r-1.ch:8080/");'

$ NODE_DEBUG=net node -e 'fetch ("http://ds-test.r-1.ch:8080/");'
NET 261750: createConnection [
  {
    highWaterMark: 65536,
    path: undefined,
    localAddress: null,
    port: '8080',
    host: 'ds-test.r-1.ch'
  },
  null,
  [Symbol(normalizedArgs)]: true
]
NET 261750: pipe false undefined
NET 261750: connect: find host ds-test.r-1.ch
NET 261750: connect: dns options { family: undefined, hints: 32 }
NET 261750: connect: autodetecting
NET 261750: connect/multiple: will try the following addresses [
  { address: '142.4.216.103', family: 4 },
  { address: '2607:5300:60:1567:1183:d3a4:263c:fb36', family: 6 }
]
NET 261750: connect/multiple: attempting to connect to 142.4.216.103:8080 (addressType: 4)
NET 261750: connect/multiple: setting the attempt timeout to 250 ms
NET 261750: connect/multiple: connection to 142.4.216.103:8080 timed out
NET 261750: connect/multiple: attempting to connect to 2607:5300:60:1567:1183:d3a4:263c:fb36:8080 (addressType: 6)
NET 261750: destroy
NET 261750: close
NET 261750: close handle
node:internal/deps/undici/undici:13178
      Error.captureStackTrace(err);
            ^

TypeError: fetch failed
    at node:internal/deps/undici/undici:13178:13
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  [cause]: AggregateError [ETIMEDOUT]: 
      at internalConnectMultiple (node:net:1116:18)
      at internalConnectMultiple (node:net:1184:5)
      at Timeout.internalConnectMultipleTimeout (node:net:1710:5)
      at listOnTimeout (node:internal/timers:583:11)
      at process.processTimers (node:internal/timers:519:7) {
    code: 'ETIMEDOUT',
    [errors]: [
      Error: connect ETIMEDOUT 142.4.216.103:8080
          at createConnectionError (node:net:1646:14)
          at Timeout.internalConnectMultipleTimeout (node:net:1705:38)
          at listOnTimeout (node:internal/timers:583:11)
          at process.processTimers (node:internal/timers:519:7) {
        errno: -110,
        code: 'ETIMEDOUT',
        syscall: 'connect',
        address: '142.4.216.103',
        port: 8080
      },
      Error: connect ENETUNREACH 2607:5300:60:1567:1183:d3a4:263c:fb36:8080 - Local (:::0)
          at internalConnectMultiple (node:net:1180:16)
          at Timeout.internalConnectMultipleTimeout (node:net:1710:5)
          at listOnTimeout (node:internal/timers:583:11)
          at process.processTimers (node:internal/timers:519:7) {
        errno: -101,
        code: 'ENETUNREACH',
        syscall: 'connect',
        address: '2607:5300:60:1567:1183:d3a4:263c:fb36',
        port: 8080
      }
    ]
  }
}

Node.js v22.6.0

@ShogunPanda

cc @ShogunPanda

@notr1ch

@notr1ch You can use --network-family-autoselection-attempt-timeout or net.setDefaultAutoSelectFamilyAttemptTimeout to adjust the default value.
Does this solve the issue for you?

Yes, increasing the timeout or using --no-network-family-autoselection works around the issue, however the default settings should establish a connection successfully. An ideal happy eyeballs implementation should not discard the first connection attempt after a timeout, but start a new one and race them both to completion.

Unfortunately we can't do that due to some limitation in our internal architecture.
Closing this.

No modern language runtime should abandon a connection after 250ms and fail in this fashion. A quick search of past closed issues and Google shows many cases of users running into this bug in real-world situations. If you can't implement proper happy eyeballs then the default timeout needs adjusting to match real-world network conditions - I would suggest a minimum of 4-5 seconds to allow for a TCP SYN to be retried to account for packet loss.

I think that is not going to work. 5 seconds before failing a connection would stall your process. This will be not acceptable for people with problematic routing.

In Node we aim to provide good defaults for the vast majority of people. Excluding some early stage problem in my first implementation of this feature, I think we are now in a good shape. Moreover, since all you need is to use a simple API call when booting your process, I think we are fine the way we are now.

Defaults matter. I opened this issue because it's happening in our production environment on a 70ms RTT API with datacenter to datacenter connectivity after we upgraded to Node v20. The internet isn't perfect, packet loss happens - a single lost packet should not cause Node to fail the connection due to the connect timeout being shorter than TCP's retry mechanisms. Having to ask people using Node 20+ to add command line parameters to make connections function normally suggests the defaults are not good.

The default also causes issues for many poorly connected countries, China in particular has very high latency through the GFW and countries like Australia have high RTT to endpoints hosted in North America and Europe. Entire types of connections will run into the issue with this default - satellite, in-flight Wi-Fi, old generation cellular networks, etc. will all regularly have over 250ms of overhead and hit this bug. Not everyone in the world is in on gigabit fiber connecting to us-east-1. And not every user of Node is a developer who will know to look through GitHub issues to find a command line parameter to fix it.

I think that is not going to work. 5 seconds before failing a connection would stall your process. This will be not acceptable for people with problematic routing.

I also don't understand how the process would stall with a 5 second timeout - as Node is event based, there should be no blocking socket operations that stall the process. As Node already uses a long timeout on the final candidate, process stalling should have already been an issue with the current implementation.

Regardless of the above arguments, I still consider the behavior described a bug. Node intentionally uses a longer timeout on the final candidate in an attempt to ensure the connection succeeds even if all other candidates have failed. It just so happens that the final candidate is an address family that isn't reachable at all in an IPv4-only environment so the intended longer timeout never even gets attempted. If a better default timeout can't be implemented, I would suggest using the long timeout on the final candidate from each address family instead of just the final candidate overall. This would ensure both IPv4 and IPv6 endpoints can be reached in the event the 250ms timeout is too short for an initial candidate to be viable.

Defaults matter. I opened this issue because it's happening in our production environment on a 70ms RTT API with datacenter to datacenter connectivity after we upgraded to Node v20. The internet isn't perfect, packet loss happens - a single lost packet should not cause Node to fail the connection due to the connect timeout being shorter than TCP's retry mechanisms. Having to ask people using Node 20+ to add command line parameters to make connections function normally suggests the defaults are not good.

I agree that default matters.
For people with limited understanding of how networks and dual stack work, 250ms timeout should ensure a good developer experience.
For other people which might have some specific needs, we provide a simple to use single call API that they can use to bypass the problem. Or a CLI when you don't control the source code.
May I ask you why you can't use the API to increase the timeout at the boot of the process?

The default also causes issues for many poorly connected countries, China in particular has very high latency through the GFW and countries like Australia have high RTT to endpoints hosted in North America and Europe. Entire types of connections will run into the issue with this default - satellite, in-flight Wi-Fi, old generation cellular networks, etc.

I totally agree on that. That's why we use a default timeout which is 2.5 higher that the minimum default which the reference standard suggests.

will all regularly have over 250ms of overhead and hit this bug. Not everyone in the world is in on gigabit fiber connecting > to us-east-1.

Neither do I. I live in the middle of nowhere Italy, definitely far away from meaningful PoP.

And not every user of Node is a developer who will know to look through GitHub issues to find a command line parameter to fix it.

That's probably a good point. Shall we include a better error when all attempts fail? Are you willing to send a PR?

I also don't understand how the process would stall with a 5 second timeout - as Node is event based, there should be no blocking socket operations that stall the process. As Node already uses a long timeout on the final candidate, process stalling should have already been an issue with the current implementation.

System resources are finite. If each connection takes up to 5 seconds, you might run of available sockets.
The choice on the final attempt was done to minimize breakage with the previous status quo, but nobody said it's ideal.

Regardless of the above arguments, I still consider the behavior described a bug. Node intentionally uses a longer timeout on the final candidate in an attempt to ensure the connection succeeds even if all other candidates have failed. It just so happens that the final candidate is an address family that isn't reachable at all in an IPv4-only environment so the intended longer timeout never even gets attempted. If a better default timeout can't be implemented, I would suggest using the long timeout on the final candidate from each address family instead of just the final candidate overall. This would ensure both IPv4 and IPv6 endpoints can be reached in the event the 250ms timeout is too short for an initial candidate to be viable.

You can provide a PR which will increase the default attempt timeout. I wouldn't go higher than a second. So we can have a broader audience and see what people think about it.

Developing a newspaper website (not the biggest in its market, but not insignificant either).
Site (on Google cloud infrastructure) uses both server-side and client-side rendering, so both backend and frontend code makes requests to the same API, which runs as a service exposed with both IPv4 and IPv6 addresses. Since we use some of G cloud's auto-scaling features, request times should be reasonable overall but sometimes might be slow.

We've been stuck running the site on Node 16 because each time we rolled out an update going to Node 18 or 20 and switching to using native fetch(), we saw random 500 errors and had user complaints that pages randomly failed to load. Debugging exposed ETIMEDOUT error messages when requesting content from the API, and I assume we're seeing the problem with the limited timeout discussed in this issue. I don't know enough about networking and this issue to really understand why we might sometimes hit it when code running on one Google cloud machine is going to talk to another Google cloud machine, but I haven't found anything else that seems equally relevant.

This has been a time consuming and frustrating problem, we have rolled out updates that we had to revert twice - the second time we even attempted to assure ourselves by doing performance testing on a staging site and observe site behaviour under load, but apparently we still hit timeouts in production eventually.

It is not clear to me from the above discussion whether any change to the default timeout has been made. I might attempt the NODE_OPTIONS workaround described here, but I would rather not since I don't know if such non-default config will introduce other problems down the road. Also, default values and algorithms should preferably "just work"..

I might attempt the NODE_OPTIONS workaround described here, but I would rather not since I don't know if such non-default config will introduce other problems down the road.

For your situation, I would recommend using --no-network-family-autoselection. This will revert to the behavior you previously had in Node 16.

net: increase network family autoselection timeout to 500ms

feat(postcraft): bump to v0.3.1-alpha.1

[API] Fix: intermittent login 401 on high-latency links (Node happy-e…

fix(net): pass a generous autoSelectFamilyAttemptTimeout

Fix Node fetch hang against Apollo via IPv4-only undici dispatcher

fix(frontend): disable autoSelectFamily, not just DNS order, to fix G…

fix: survive happy-eyeballs connect timeouts and surface network erro…

Data point from Algeria (high-RTT consumer ISP, intermittently broken IPv6), for the record on why 250ms - and possibly 500ms - is too short in practice:

DNS for Google endpoints rotates between edges; on far-edge days the IPv4 connect RTT alone exceeds 250ms.
The ISP router intermittently advertises a dead IPv6 default route, so the IPv6 attempt burns its slot with ENETUNREACH.
Result on Node 22 (default 250ms): net.connect emits AggregateError [ETIMEDOUT] for every dual-stack host while curl on the same machine succeeds. The failure comes and goes with DNS edge rotation, which makes it look like application flakiness rather than a connect-budget problem.

Two observations beyond the timeout value itself:

The AggregateError carries an empty message, so any fetch wrapper that only forwards error.message (node-fetch's FetchError is the common one: request to <url> failed, reason: with nothing after reason:) hides the diagnosis completely. Until parallel attempts land, a non-empty summary message on the aggregate (e.g. codes per family) would make this class of failure vastly easier to identify in the wild.
net: increase network family autoselection timeout to 500ms #60334's 500ms ships in v25.2+ only; every current LTS line keeps 250ms. If backporting the default change is off the table, LTS users hit this until 2027+. We now ship net.setDefaultAutoSelectFamilyAttemptTimeout(2000) as a library default (skipped when the user sets the flag), which resolved it for us, for what that data point is worth when weighing the default.

RFC 8305 parallel attempts (#48145) obviously remains the real fix.

Backend: fix "fetch failed" on slow networks, log the real network cause

@peat-psuwit

Another data point, since this isn't limited to high-RTT links: we hit it in CI on a datacenter runner without an IPv6 route, using fetch from actions/github-script on Node 24 to reach a Google Cloud load balancer with both A and AAAA records. The IPv4 handshake took longer than 250ms, most likely because of a single dropped SYN, and the IPv6 attempt then failed immediately with no route, so the request failed with AggregateError [ETIMEDOUT] after about 250ms.

@peat-psuwit's suggestion above would have avoided this: once the last candidate fails with ENETUNREACH, the attempt Node abandoned was the only viable one. We worked around it with net.setDefaultAutoSelectFamilyAttemptTimeout(2000) plus retries.

Navigation Menu

Repository navigation

Happy eyeballs implementation times out prematurely #54359

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior? Why is that the expected behavior?

What do you see instead?

Additional information

13 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Happy eyeballs implementation times out prematurely #54359

Description

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior? Why is that the expected behavior?

What do you see instead?

Additional information

Activity

avivkeller commented on Aug 13, 2024

notr1ch commented on Aug 13, 2024

notr1ch commented on Aug 13, 2024

avivkeller commented on Aug 13, 2024

marco-ippolito commented on Aug 14, 2024

ShogunPanda commented on Aug 14, 2024

notr1ch commented on Aug 14, 2024

ShogunPanda commented on Aug 14, 2024

notr1ch commented on Aug 14, 2024

ShogunPanda commented on Aug 15, 2024

notr1ch commented on Aug 15, 2024

ShogunPanda commented on Aug 16, 2024

13 remaining items

hallvors commented on Oct 9, 2025

notr1ch commented on Oct 10, 2025

bakissation commented on Sep 19, 2026

mvdan commented on Oct 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions