Repository navigation
Performance of 'http' + anonymous functions eventually slows down #3659
Description
When using anonymous functions (in IIFEs or Promises) to prepare the response of an HTTP request, the performance of the server suddenly slows down. Performance remains low until the server is restarted.
I initially found the problem using Promises. A script to replicate the problem:
var http = require('http');
var srv = http.createServer(function (req, res) {
promiseValue().then(function(value){
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end(value);
});
});
function promiseValue() {
return Promise.resolve().then(function() {
var j = 0;
for (var i = 0, l = 300000; i < l; i++) { j = j + 2; }
return "Hi " + j;
});
}But Promises are not required:
var http = require('http');
var srv = http.createServer(function (req, res) {
callbackValue(function(value){
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end(value);
});
});
function callbackValue(cb) {
(function() {
var j = 0;
for (var i = 0, l = 300000; i < l; i++) {
j = j + 2;
}
cb("Hi " + j);
})();
}
srv.listen(3000, '127.0.0.1', function() {
"use strict";
console.log("Server listening on 127.0.0.1:3000");
});Using a HTTP benchmarking tool to test the server, e.g. wrk:
#! /bin/bash
while :
do
wrk -t2 -c32 -d5s "http://127.0.0.1:3000" | grep "Requests"
doneThe server starts providing good performance, but an unpredictable point in time the performance drops:
Requests/sec: 2389.40
Requests/sec: 2394.00
Requests/sec: 2396.53
Requests/sec: 1071.95
Requests/sec: 890.80
Requests/sec: 889.23
Requests/sec: 893.15
Once it drops, it doesn't matter if you stop the load testing tool and let the server recover for a while, performance remains low until the server is restarted.
¿Is it a bug or am I doing something wrong?
If it is a bug, I don't know if it belongs to Nodejs or V8, although I have been unable to reproduce the problem only with Promises/IIFEs without the 'http' module.
Some additional details I have collected while hunting down the issue that may or may not be of interest to solve the problem:
- It can be reproduced in all versions of Nodejs, at least in OSX. In particular, using Promises, I have been able to reproduce it in: v5.0.0, v4.2.1, iojsv1.0.0, even v0.11.13 (the first version that introduced Promise). Previous versions (v10.x, <v0.11.13) didn't have native Promise support, I have tested them using a 3rd party library (Bluebird) and performance matches the low one since the start up.
- Performance drops after an unpredictable amount of time or requests. Following the same steps, I have found the drop in performance in a wide range of time (e.g. 15 seconds - 30 minutes).
- The anonymous function used in the
.then(function(){...})ofpromiseValue()is the main cause of performance. - According to its performance, during the 1st phase the function seems to be optimized and during the 2nd phase it seems to become deoptimized (see comment).
- If the anonymous function is refactorized as a named function, performance matches the high level and does NOT drop over time.
- If an instruction that prevents optimization (e.g.
try {} catch (e) {}) is added to the anonymous function, performance matches the low level and doesn't vary over time.
- According to
v8, the functionis always not optimized(I could copy the snippet of how I tested this if required. Basically, it follows @petkaantonov's tip). (EDIT: The function starts being optimized and, at some point, becomes deoptimized also according tov8, see comment).
(I can also provide snippets for these points)
EDIT:
Activity
[-]Performance of 'http' + anonymous functions eventually slowdown[/-][+]Performance of 'http' + anonymous functions eventually slows down[/+]alexlamsl commentedon Nov 4, 2015
Might not be related, but can you run
netstaton the command line and see if you get a lot of stale TCP connections over the course of the test?YurySolovyov commentedon Nov 4, 2015
@alexlamsl but that would not explain why it performs ok without IIFE or promises.
ChALkeR commentedon Nov 4, 2015
@SynerG Could you also log server memory usage?
@SynerG Also,
--trace-gc.https://code.google.com/p/v8/issues/detail?id=4121 could be related.Edit: If 0.11.13 is affected, then it shouldn't.
Ok, I ran it with
--trace-gc, and it's not the culprit. Memory usage stays at ~50 MiB, gc times are low.SynerG commentedon Nov 4, 2015
@YuriSolovyov, I think you could also reproduce the issue in your machine, but you will probably need to run the script for a longer period of time.
I have just tried it in a Linux VM and in a droplet in Digital Ocean and both have eventually degraded, but it has required several minutes of the server+script running.
From the DO VM:
@alexlamsl, I have run
netstat -anin the Linux VMs after performance has degraded (stopping the load testing tool, but leaving the server open) and it doesn't seem to leave open connections. I check the state after stoppingwrkto see if there are connections left, because when I runwrkagain for the second time, performance remains already degraded since the beginning.@ChALkeR, regarding memory usage:
GC show lines like these:
Also, the state of node process according to
top:So it looks like the implementation creates a memory leakage, although the VM still has free memory even after performance has degraded.
YurySolovyov commentedon Nov 4, 2015
I wonder if you can build node with this and try again, just in case.
SynerG commentedon Nov 4, 2015
@YuriSolovyov, I am not familiar with the process to rebuild Nodejs with a different version of v8.
I guess I have to:
Is this the correct process?
targos commentedon Nov 4, 2015
@SynerG It is not the easiest process. First you can try to directly apply the patch.
Download it: https://github.com/proxy/gist.github.com/targos/b057bbaf1fd1589d4dba
And from the nodejs repo, run
git apply 4dd45e115be0122ee8b6ba1d9b3be5e86ea3b97c.diff --directory=deps/v8If you are lucky, it will apply cleanly
kzc commentedon Nov 4, 2015
Since the problem is 100% reproducible given the excellent test case @SynerG provided any contributer could test out any patch they wish.
evanlucas commentedon Nov 4, 2015
That patch does actually seem to prevent the large drop, but the average seems to be a bit lower right off the bat.
on the master branch:
Before
After
YurySolovyov commentedon Nov 4, 2015
@evanlucas I was blinded by this too, try to run it at least 5-10 mins.
SynerG commentedon Nov 4, 2015
@targos thanks for the tip. I was already rebuilding node the hard way when I read your comment.
I have rebuild node with the latest version of v8 in the repo. It should already contain the desired commit as there is a path between the two according to git:
In my OSX development machine, the patch doesn't solve the problem:
@evanlucas could you keep running the test longer? It could eventually reach the point when it drops yet
kzc commentedon Nov 4, 2015
Another data point - I tried out the (no Promise) test case on Mac with node v5.0.0 and got the same throughput drop off numbers as @SynerG at top of ticket: 2400 requests/sec => 900 requests/sec after 5 minutes.
Then for the heck of it, I tried running the test case on jxcore, a node port using the mozilla JS engine - not v8:
After an hour I did not see any drop off:
which suggests it may be a v8 issue.
SynerG commentedon Nov 4, 2015
@kzc good point using
jxcore. Do they only replace the JS engine or could they have also reimplemented node features such as thehttpmodule?It may be a v8 issue, but I have been unable to reproduce the problem using only v8 features.
I have tried to remove the
httpmodule and replace it with continuous calls tocallbackValue(), but performance doesn't degrade over time.For example, using the following script:
The snippet doesn't use
http, but performance remains high even after almost 2 hours running:This alternative to replace
httpalso fails to reproduce the issue (i.e. performance remains high after an hour):If we were able to come up with a snippet that triggered the problem without
http, using only v8 features, we could run the script in Chrome/Chromium, without Nodejs, and reproduce the issue. We could determine that the problem belongs to v8.Is somebody able to come up with a good script for this?
31 remaining items
trevnorris commentedon Nov 9, 2015
This is pretty much what we should be doing. On every object instantiation we should be setting all possible future fields to at least
null. This is enough to prevent object map mutation.jasnell commentedon Apr 9, 2016
Ping. any updates on this one?
targos commentedon Jan 2, 2017
I ran the test several times with v5, v6 and v7 and can only reproduce the performance drop in v5.
I believe the issue was fixed by a V8 update, so I'm going to close this.
MylesBorins commentedon Jan 2, 2017
@targos did you measure v4?
targos commentedon Jan 2, 2017
I didn't but it's probably still affected
MylesBorins commentedon Jan 2, 2017
would we be able to backport the V8 changes?
targos commentedon Jan 2, 2017
We'd have to identify them. I reopen with the
v4.xlabel in case someone wants to do it.YurySolovyov commentedon Nov 15, 2017
Still the case on node 8.3+ ?
apapirovski commentedon Dec 6, 2017
Given that Node.js 4.x is now in maintenance mode (and has less than 5 months to go in its life) and there has been absolutely zero interest in anyone taking this on, I'm going to go ahead and close it but feel free to reopen if you believe I've made an error.