Repository navigation
test-osuosl-aix61-ppc64_be build failures #494
Description
test-osuosl-aix61-ppc64_be-2 seems to be reliably failing to build for the last 24 hours or so.
Example failure: https://ci.nodejs.org/job/node-test-commit-aix/936/nodes=aix61-ppc64/console
@jbergstroem said in IRC:
a lot of hanging test failures:
this: /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/node /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/parallel/test-child-process-fork-dgram.js child
..and this: /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/node /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/sequential/test-child-process-pass-fd.js child
seems to be a few 'wait' processes. I wonder if that syntax behaves differently on aix
been active for four days :( i'm on the phone for a couple of hours still -- could you perhaps file an issue and ping me/mhdawson?)
(we should ping gibfahn too -- he had a few tests that didn't exit either)
Activity
gibfahn commentedon Sep 20, 2016
I'll take a look, also cc/ @gireeshpunathil
jbergstroem commentedon Sep 20, 2016
Just a note: I killed all lingering processes on both machines since the test runner was stalling.
mhdawson commentedon Sep 20, 2016
I don't see any failures today since @jbergstroem comment and I also had to restart the jenkins agent. Since then its seems to be be running ok. We should probably check tomorrow to see if there are any jobs stacking up in the background.
mhdawson commentedon Sep 21, 2016
Looked today. There were a small number of processes hanging around:
gibfahn commentedon Sep 21, 2016
@mhdawson That's interesting, I hadn't realised that the child processes weren't being killed in that test failure. So if the test times out, I assume the test runner kills the parent but not any children?
jbergstroem commentedon Sep 21, 2016
@mhdawson if you check my above paste (well, Rich's)
test/sequential/test-child-process-pass-fd.jsseems affected too.mhdawson commentedon Sep 21, 2016
I've been looking at the fork-dgram failure, will just about to test proposed fix no AIX now. The other one must be much less frequent.
mhdawson commentedon Sep 21, 2016
PR to address fork-dgram failures nodejs/node#8697
[-]test-osuosl-aix61-ppc64_be-2 build failures[/-][+]test-osuosl-aix61-ppc64_be build failures[/+]jbergstroem commentedon Sep 23, 2016
@mhdawson, @gibfahn: log into -1 now and have a look. lots of stuff stalling. Here's a job that's been going for two hours as well (probably as a result of multiple test runners still being active): https://ci.nodejs.org/job/node-test-commit-aix/1040/nodes=aix61-ppc64/console
gibfahn commentedon Sep 23, 2016
@jbergstroem I'm not sure whether it's because someone's cleaned up the machine (or because
ps -efisn't the right command), but I'm only seeing a couple of test-dgram processes on that machine.Also when I first clicked your link I saw the still-running job (and I still have it open in a tab):
But now when I click through I don't:
jbergstroem commentedon Sep 23, 2016
@gibfahn you also have a few python processes:
..and a few gmake processes:
This is probably not good seeing how each test.py makes the assumption that they will be run exclusively (tmpdir, whatnot).
jbergstroem commentedon Sep 23, 2016
You can also see the
waitinvoked as well as as few defunct processes.gibfahn commentedon Sep 23, 2016
@jbergstroem I'm not seeing those on
test-osuosl-aix61-ppc64_be-1(a.k.a.power8-nodejs2.osuosl.org). Are you looking at-2?EDIT: I do see them on
-2jbergstroem commentedon Sep 23, 2016
@gibfahn if I am then there's a naming thing. I'm talking about
140.211.9.101.gibfahn commentedon Sep 23, 2016
I might have them the wrong way round in my
.ssh/config, I have:Anyway, I'll take a look at why the processes are being left behind.
mhdawson commentedon Oct 20, 2016
Ok, the test that was causing processes to still be running after the test run was fixed so I think we can close this. Please re-open if you feel differently.