feat(relay): splice(2) on Linux for the none cipher and bypassed connections

[pylaterreur]

With the none method, a proxied TCP connection carries the application's bytes unchanged after the address header, and so does every connection that the ACL sends around the proxy. sslocal and ssserver still read these bytes into a userspace buffer and write them out again. On Linux and Android, this PR moves them from one socket to the other through a pipe with splice(2) instead, so they never leave the kernel. Go's net package does the same for io.Copy between TCP connections, which is how shadowsocks-go relays none.

Nothing changes on the wire, for proxied connections with other methods, or on other systems.

When the relay splices

• sslocal, none (first commit): after the first packet (address and first payload) is out, if the connection to the server is a plain TcpStream (not an outbound proxy chain, not TCP Fast Open) and the client side is one too (tunnel, SOCKS, redir). The TUN stack and hyper's upgraded HTTP connections keep the current path.
• ssserver, none (first commit): after the handshake has read the address, which it does byte by byte and so leaves everything behind it in the socket, if the connection to the target is direct.
• sslocal, bypassed connections (second commit): any method, if both sides are plain TcpStreams. These are plain TCP relays today, through tokio's copy_bidirectional with 8 KiB buffers.

Everything else takes the existing path.

Numbers

sslocal (tunnel) → ssserver → target, 2 GB per transfer, every transfer checked byte for byte, on loopback. The leg between sslocal and ssserver was plain TCP, then MPTCP (--tcp-multipath). Linux 6.17 on a Ryzen 9 5900X (8-vCPU VM), 5 rounds with the builds alternating within each round, medians. The machine was shared with other VMs (1-minute load 1.6 to 7.3, median 2.9), so throughput varies a lot between runs; the CPU per GB is steadier. CPU is the seconds per GB of each process, lower is better. Method none:

leg	direction		master	this PR	change
TCP	upload	sslocal CPU s/GB	1.48	0.38	−75 %
		ssserver CPU s/GB	1.28	0.33	−74 %
		throughput Gbit/s	5.40	11.19	+107 %
	download	sslocal CPU s/GB	1.95	0.33	−83 %
		ssserver CPU s/GB	1.19	0.34	−72 %
		throughput Gbit/s	4.47	12.13	+171 %
MPTCP	upload	sslocal CPU s/GB	2.71	0.45	−83 %
		ssserver CPU s/GB	1.61	0.81	−50 %
		throughput Gbit/s	3.00	9.69	+223 %
	download	sslocal CPU s/GB	1.96	0.59	−70 %
		ssserver CPU s/GB	3.62	1.20	−67 %
		throughput Gbit/s	1.87	5.21	+179 %

On 6.17, splicing out of an MPTCP socket still copies inside the kernel, so the side that reads from MPTCP gains less (ssserver on upload, sslocal on download).

Compared with the 64 KiB buffers of #2198 alone (the userspace copy at its best), splice takes another 59 % (upload) and 64 % (download) off sslocal's CPU per GB, and 52 % and 54 % off ssserver's, with plain TCP.

Bypassed connections, measured with an sslocal tunnel that has no server configured, so it relays directly to the target through the same establish_tcp_tunnel_bypassed:

direction		master	this PR	change
upload	sslocal CPU s/GB	2.21	0.27	−88 %
	throughput Gbit/s	3.84	13.78	+259 %
download	sslocal CPU s/GB	2.48	0.26	−89 %
	throughput Gbit/s	3.50	13.90	+297 %

2022-blake3-aes-256-gcm, which keeps the existing path, measured the same within the noise: the medians of the per-round ratios to master were 0.88 to 1.06 for the CPU of either process.

How it works

crates/shadowsocks/src/relay/tcprelay/splice.rs, #[cfg(any(target_os = "linux", target_os = "android"))], no new dependency:

• copy_bidirectional_splice(a, b) has the semantics of tokio::io::copy_bidirectional. EOF on one side shuts the other down for writing, and the opposite direction keeps going. The result is the byte count of each direction. A socket whose peer has reset counts as shut down (ENOTCONN), as in tokio.
• Each direction gets its own pipe, created when its source first becomes readable, so a direction that carries nothing takes none: pipe2(O_NONBLOCK | O_CLOEXEC), resized with F_SETPIPE_SZ on a best-effort basis (see the trade-offs below). Readiness goes through readable()/writable() and TcpStream::try_io, so an EAGAIN from splice clears tokio's readiness.
• Fallback. If splice fails with EINVAL, ENOSYS or EOPNOTSUPP before a direction's first successful splice into the destination, that direction carries on with a userspace copy. The bytes already in the pipe are written out first, and a failed splice consumes nothing, so nothing is lost. The pipe is closed then. A failed pipe2, or a pipe smaller than 64 KiB (what the kernel gives an unprivileged user over their pipe budget), means copying as well.
• Urgent data. splice from TCP stops in front of urgent (MSG_OOB) data and keeps returning EAGAIN (or 0 once the FIN is in) for as long as nothing reads past it, which would stall the direction. When it returns EAGAIN or 0 on a TCP socket, a one-byte recv(MSG_PEEK) checks whether recv would return data. If so, and a second splice still moves nothing (so the bytes didn't simply arrive in between), recv reads past the urgent byte. It drops that byte, as the userspace copy does. In a C relay built the same way, the peek cost nothing measurable: 0.159 vs 0.160 CPU s/GB, 30 runs each.
• SPLICE_F_MORE is not set: it would cork the last segment of a transfer.
• SIGPIPE. splice can't take MSG_NOSIGNAL, so splicing into a socket whose peer is gone raises SIGPIPE. Rust executables ignore it. If the library runs in a process where SIGPIPE has its default action (checked with sigaction), the relay copies through userspace instead of splicing.
• Flow statistics. The spliced bytes bypass MonProxyStream, so they are reported to the connection's FlowStat through a small SpliceMonitor trait. Bypassed connections have no flow statistics, as before.
• Plumbing. ProxyClientStream::passthrough_ref() and ProxyServerStream::passthrough_ref() return the inner stream only for none once the handshake is done. In shadowsocks-service, a SpliceIo trait exposes the socket behind a stream; its default is no socket, which is always correct. TcpStream::as_tokio() returns None for TCP Fast Open streams.

Trade-offs

• File descriptors. A spliced connection has two pipes, so four more file descriptors. A failed pipe2 (EMFILE, for example) means copying, as today.

• Pipe size: 256 KiB per direction, on a best-effort basis. A pipe holds what the destination hasn't taken yet, so this is the most a direction keeps in the kernel while the destination is slower than the source. Unprivileged processes, such as the units in debian/ (DynamicUser=yes), get 64 MiB of pipes in all by default (fs.pipe-user-pages-soft). Past that, new pipes get two pages, and those connections copy as today. That allows 128 connections spliced both ways at 256 KiB, against 32 at the 1 MiB that Go uses. Medians of the per-round ratios to 1 MiB, from the same runs:

  	256 KiB	64 KiB (kernel default)
  sslocal CPU, TCP upload / download	0.97 / 1.09	1.19 / 1.34
  ssserver CPU, TCP upload / download	1.00 / 0.99	1.20 / 1.11
  sslocal CPU, MPTCP leg upload / download	1.05 / 1.04	1.55 / 1.22
  throughput, MPTCP leg upload / download	1.03 / 1.00	0.77 / 0.75
  sslocal CPU, bypassed upload / download	1.06 / 0.99	1.30 / 1.26

  A pipe slot holds one fragment of the socket's data. On loopback the fragments are large, so these runs favour small pipes. Behind a real NIC a fragment is typically about one MTU of payload, so 64 slots would still move about 90 KiB per splice, which is more than one read of the 64 KiB userspace copy. The size is one constant (PIPE_CAPACITY) if you'd rather have another default or an option.

• Memory. While a connection is spliced, its userspace relay buffers aren't allocated. When the source is TCP, the pages in a pipe are the ones the socket received, not copies.

Tests

• Unit tests in splice.rs, against real loopback sockets:
  • both directions with byte counts; empty and 1-byte transfers; half-close with an answer sent after the EOF;
  • a slow reader (EAGAIN on the write side);
  • each fallback errno on either splice, including bytes left in the pipe; the same errno after a success is an error; the pipe is released on fallback;
  • MPTCP on either side; urgent data mid-stream and right before the FIN; data that arrives between an EAGAIN and the peek;
  • ENOTCONN on shutdown after a reset, and a reset during the transfer;
  • SIGPIPE with its default action, checked in a forked child.
• Integration tests in tests/none_splice.rs, sslocal (tunnel) + ssserver in-process. They check every byte, check which path carried the bytes (from the relay's trace log), and check sslocal's flow statistics. The cases: 64 MiB each way with half-close; TCP Fast Open (sslocal copies, ssserver splices); aes-256-gcm (never spliced); MPTCP; full duplex; an idle client; a server that speaks first; payload sent in the same segment as the header; 64 parallel connections; a SOCKS5 connection bypassed by the ACL.
• Mutation checks, on an earlier revision of this branch: removing the shutdown at EOF, losing the pipe's bytes on fallback, or swapping the flow-statistics directions each makes tests fail.
• Run on Linux 6.17 with Rust 1.99.0: cargo test of the workspace with default features (full) and with --no-default-features, and of both crates with default and no features; cargo build of crates/shadowsocks with all its cipher features; clippy with CI's flags, with and without --all-targets. No new warnings. Not run here: the full-extra tests and the all-features cargo build (no disk space left), a build with the 1.91 MSRV (clippy's incompatible_msrv, which reads msrv = "1.91" from clippy.toml, reports nothing), macOS and Windows.
• The non-Linux code paths were compiled and tested on Linux with the cfg conditions inverted, on the revision before the pipe size changed, which is the only difference. Android builds the same module (net/sys/unix/linux), and libc 0.2.190 defines every function and constant it uses for Android. I couldn't run cargo check --target aarch64-linux-android here: blake3's build script needs the NDK's clang.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GwUvhmzgfkiZ6vo4uGyesN