Hetzner Benchmarking Machine Replacements

The procurement process has completed, and I have created two EX44's (https://www.hetzner.com/dedicated-rootserver/ex44/) at Hetzner.

If all goes well we should be able to have these online and running benchmark tests.

I believe the next steps are

Get access to the secrets repo
Configure machines with ansible
Get admin for the ci.nodejs.org to add them to jenkins
test them

@mcollina

@mcollina is it ok for performance measurements to have a CPU with a mix of non-identical cores?

It should be possible to schedule certain processes only on a subset of the cores with taskset.

This should be part of the testing phase.

These machines have been provisioned and added to Jenkins with the same labels/configs as the former nearform machines.

Next steps: Can somebody with permssions can kick off a benchmarking jobs and some v8 builds to verify that all is working as intended?

I've marked the two Nearform machines offline in Jenkins and started a V8 build which is running on test-hetzner-ubuntu2204-x64-1:
https://ci.nodejs.org/job/node-test-commit-v8-linux/5876/nodes=benchmark-ubuntu2204-intel-64,v8test=v8test/

I've marked the two Nearform machines offline in Jenkins and started a V8 build which is running on test-hetzner-ubuntu2204-x64-1: https://ci.nodejs.org/job/node-test-commit-v8-linux/5876/nodes=benchmark-ubuntu2204-intel-64,v8test=v8test/

This has failed:

02:47:26 + DEPOT_TOOLS_DIR=/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools
02:47:26 + PATH=/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools:/home/iojs/build/workspace/node-test-commit-v8-linux/depot_tools:/home/iojs/venv/bin:/home/iojs/nghttp2/src:/home/iojs/wrk:/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin tools/dev/v8gen.py x64.release --no-goma
02:47:26 
02:47:26 Hint: You can raise verbosity (-vv) to see the output of failed commands.
02:47:26 
02:47:26 Traceback (most recent call last):
02:47:26   File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 309, in <module>
02:47:26     sys.exit(gen.main())
02:47:26   File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 303, in main
02:47:26     return self._options.func()
02:47:26   File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 162, in cmd_gen
02:47:26     self._call_cmd([
02:47:26   File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 211, in _call_cmd
02:47:26     output = subprocess.check_output(
02:47:26   File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
02:47:26     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
02:47:26   File "/usr/lib/python3.10/subprocess.py", line 526, in run
02:47:26     raise CalledProcessError(retcode, process.args,
02:47:26 subprocess.CalledProcessError: Command '['/usr/bin/python3', '-u', 'tools/mb/mb.py', 'gen', '-f', 'infra/mb/mb_config.pyl', '-m', 'developer_default', '-b', 'x64.release', 'out.gn/x64.release']' returned non-zero exit status 1.
02:47:26 make: *** [Makefile:303: v8] Error 1

Logging into the machine and running the failing command with -vv (as suggested):

iojs@test-hetzner-ubuntu2204-x64-1:~/build/workspace/node-test-commit-v8-linux/deps/v8$ PATH=/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/_depot_tools:/home/iojs/build/workspace/node-test-commit-v8-linux/depot_tools:/home/iojs/venv/bin:/home/iojs/nghttp2/src:/home/iojs/wrk:/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin tools/dev/v8gen.py x64.release --no-goma -vv
################################################################################
/usr/bin/python3 -u tools/mb/mb.py gen -f infra/mb/mb_config.pyl -m developer_default -b x64.release out.gn/x64.release

  Writing """\
  dcheck_always_on = false
  is_debug = false
  target_cpu = "x64"
  """ to /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/out.gn/x64.release/args.gn.

  /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/buildtools/linux64/gn gen out.gn/x64.release --check
    -> returned 1
  ERROR at //build/config/linux/pkg_config.gni:104:17: Script returned non-zero exit code.
      pkgresult = exec_script(pkg_config_script, args, "json")
                  ^----------
  Current dir: /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/out.gn/x64.release/
  Command: python3 /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/build/config/linux/pkg-config.py -s /home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/build/linux/debian_bullseye_amd64-sysroot -a x64 glib-2.0 gmodule-2.0 gobject-2.0 gthread-2.0
  Returned 1.
  stderr:

  Traceback (most recent call last):
    File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/build/config/linux/pkg-config.py", line 247, in <module>
      sys.exit(main())
    File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/build/config/linux/pkg-config.py", line 142, in main
      prefix = GetPkgConfigPrefixToStrip(options, args)
    File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/build/config/linux/pkg-config.py", line 80, in GetPkgConfigPrefixToStrip
      prefix = subprocess.check_output([options.pkg_config,
    File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/usr/lib/python3.10/subprocess.py", line 503, in run
      with Popen(*popenargs, **kwargs) as process:
    File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
      self._execute_child(args, executable, preexec_fn, close_fds,
    File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
      raise child_exception_type(errno_num, err_msg, err_filename)
  FileNotFoundError: [Errno 2] No such file or directory: 'pkg-config'

  See //build/config/linux/BUILD.gn:58:3: whence it was called.
    pkg_config("glib") {
    ^-------------------
  See //build/config/compiler/BUILD.gn:300:18: which caused the file to be included.
      configs += [ "//build/config/linux:compiler" ]
                   ^------------------------------
  GN gen failed: 1
Traceback (most recent call last):
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 309, in <module>
    sys.exit(gen.main())
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 303, in main
    return self._options.func()
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 162, in cmd_gen
    self._call_cmd([
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/deps/v8/tools/dev/v8gen.py", line 211, in _call_cmd
    output = subprocess.check_output(
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-u', 'tools/mb/mb.py', 'gen', '-f', 'infra/mb/mb_config.pyl', '-m', 'developer_default', '-b', 'x64.release', 'out.gn/x64.release']' returned non-zero exit status 1.
iojs@test-hetzner-ubuntu2204-x64-1:~/build/workspace/node-test-commit-v8-linux/deps/v8$

Looking at the other nearform benchmarking machines I see a considerable amount of manual package installations that are outside of ansible's setup.

There's at least 220 packages that have been manually installed.

Theres also the likelyhood that I was supposed to configure ansible differently to set up these machines beyond what I understood.

I attempted to add the 'is_benchmark' = true to add the benchmark role as I see that pkg-config is installed that way, but some of the other packages in that role have been dropped from ubuntu and that role hasnt been updated for six years, so its probably got some stale packages in there.

List of missing packages

apport-symptoms
aptitude
apt-transport-https
bind9-host
bison
bsdmainutils
build-essential
busybox-initramfs
bzip2
ca-certificates
clang
cloud-guest-utils
cloud-initramfs-copymods
cloud-initramfs-dyn-netconf
console-setup-linux
coreutils
cryptsetup-bin
debianutils
dh-python
distro-info-data
dmeventd
dmsetup
dnsmasq-base
dns-root-data
dnsutils
dpkg
eject
file
flex
ftp
gawk
geoip-database
gettext-base
git-man
gnutls-bin
gpgv
groff-base
grub-common
grub-legacy-ec2
grub-pc
ifenslave
ifupdown
initramfs-tools
initramfs-tools-bin
initramfs-tools-core
init-system-helpers
install-info
isc-dhcp-common
keyboard-configuration
klibc-utils
krb5-locales
language-selector-common
libacl1
libapparmor1
libatm1
libattr1
libaudit1
libaudit-common
libblkid1
libbsd0
libbz2-1.0
libc6
libcap2
libcap2-bin
libcap-dev
libc-ares-dev
libc-bin
libcunit1-dev
libcurl4
libdb5.3
libdevmapper1.02.1
libdevmapper-event1.02.1
libdumbnet1
liberror-perl
libestr0
libev-dev
libevent-dev
libexpat1
libfdisk1
libfribidi0
libgcrypt20
libglib2.0-data
libgmp10
libgnutls30
libgnutls-openssl27
libgpg-error0
libgpm2
libjansson-dev
libkeyutils1
libklibc
libkmod2
liblocale-gettext-perl
liblxc1
liblz4-1
liblzma5
liblzo2-2
libmagic1
libmnl0
libmount1
libmspack0
libncurses5
libncursesw5
libnetfilter-conntrack3
libnewt0.52
libp11-kit0
libpam0g
libpam-modules
libpam-modules-bin
libpam-runtime
libpcre3
libpolkit-agent-1-0
libpopt0
libpython3-stdlib
libreadline6
libsasl2-modules
libseccomp2
libselinux1
libsemanage-common
libsigsegv2
libslang2
libsmartcols1
libsqlite3-0
libss2
libssl-dev
libstdc++6
libsystemd0
libtasn1-6
libtext-charwidth-perl
libtext-iconv-perl
libtext-wrapi18n-perl
libtinfo5
libudev1
libusb-0.1-4
libustr-1.0-1
libutempter0
libuuid1
libwrap0
libx11-data
libxml2-dev
linux-base
linux-generic
linux-headers-generic
lsb-base
ltrace
lxcfs
makedev
mime-support
mlocate
ncurses-base
ncurses-term
ntfs-3g
openjdk-8-jre-headless
openssh-sftp-server
openssl
pastebinit
perl
perl-base
pkg-config
policykit-1
popularity-contest
powermgmt-base
python2
python3.7-distutils
python3-apport
python3-apt
python3-chardet
python3-commandnotfound
python3-dbus
python3-debian
python3-distupgrade
python3-gdbm
python3-gi
python3-minimal
python3-newt
python3-pkg-resources
python3-problem-report
python3-pycurl
python3-requests
python3-setuptools
python3-six
python3-software-properties
python3-systemd
python3-update-manager
python3-urllib3
python-apt-common
r-base
readline-common
rename
resolvconf
run-one
sed
sgml-base
shared-mime-info
snap-confine
snapd
squashfs-tools
tar
tasksel
tcpd
telnet
traceroute
ubuntu-cloudimage-keyring
ubuntu-core-launcher
ubuntu-minimal
ubuntu-standard
ucf
uidmap
unzip
util-linux
vim-common
vim-runtime
vlan
xauth
xdg-user-dirs
xkb-data
xml-core
zerofree
zlib1g
zlib1g-dev

Take a look at the above list and we should decide if we need to update ansible to include some of this setup or not.

I suspect it is the is_benchmarking variable that installed those packages, and that is has bit rotted. For example, pkg-config (the missing thing in #3657 (comment)) is listed

build/ansible/roles/benchmarking/vars/main.yml

Line 10 in 51ad778

- pkg-config

.

We should probably add pkg-config to

build/ansible/roles/build-test-v8/tasks/partials/ubuntu2204.yml

Line 9 in 51ad778

name: ['ninja-build', 'python3-httplib2']

as it appears needed to build V8.

I'm less familiar with what is needed to run the benchmarks.

I attempted to add the 'is_benchmark' = true to add the benchmark role as I see that pkg-config is installed that way, but some of the other packages in that role have been dropped from ubuntu and that role hasnt been updated for six years, so its probably got some stale packages in there.

Adding pkg-config this way has allowed the V8 CI to build and run tests.

Node.js main branch failed due to Broken test/v8-updates/test-linux-perf-logger node#51308: https://ci.nodejs.org/job/node-test-commit-v8-linux/5878/nodes=benchmark-ubuntu2204-intel-64,v8test=v8test/consoleFull
Node.js 18 branch failed due to missing perf: https://ci.nodejs.org/job/node-test-commit-v8-linux/5879/nodes=benchmark-ubuntu2204-intel-64,v8test=v8test/console

Neither encountered the networking issues we had with the Nearform hosted benchmark machines (🎉).

For the missing perf, maybe we need to add the hetzner machines to:

build/ansible/playbooks/jenkins/worker/create.yml

Lines 74 to 77 in 51ad778

    
           - hosts: 
        
               - test-*intel-ubuntu*-x* 
        
             roles: 
        
               - linux-perf

For the missing perf, maybe we need to add the Hetzner machines to:

That intel tag was what was being used to target the Nearform intel donated machines. We should definitely change that to target the Hetzner ones now.

Adding pkg-config this way

How shall we approach getting these machines into a stable usable state going forward? I can continue to adjust which packages are installed as part of the ansible setup, but I don't want to inadvertently step on or undo any work that anybody else is doing. (Though I also lack any background in what the jobs do/accomplish)

Ran the benchmark job which fails - https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1498/

Looks like there may be a directory missing. That may have been created manually as these machines were set up a long time ago.

@ryanaslett

@ryanaslett have you added the linux-perf role? That might be all that is needed to get the v8 jobs running as well as they were before on the machines. @richardlau is that your expectation?

manually creating the directory /w owned by iojs and with group iojs has let the benchmark run get further

@ryanaslett

@ryanaslett if you are updating the ansible scripts, is there a section which is specific to the benchmark machines that we can add the creation of the /w directory owned by iojs with group iojs?

Job to see if perf job runs ok after adding the /w directory - https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1499/

@ryanaslett

@ryanaslett if you are updating the ansible scripts, is there a section which is specific to the benchmark machines that we can add the creation of the /w directory owned by iojs with group iojs?

Thats sort of what I was asking in #3657 (comment) - mostly who should be doing this.

I left a comment in nodejs/performance#157 -- I think the script is currently limiting the benchmarks to a single CPU, which is probably not the intent 🙂. See nodejs/node#52233 (comment). Perhaps the script could parameterize the --set parameter and then we could set it in the Jenkins job.

Done: nodejs/performance#158

@rluvaton

@rluvaton btw we can use nproc command to max cores instead of deafult 0-11 in the script

nproc doesn't know the difference between performance and efficiency cores.

@rluvaton

@rluvaton based on #3657 (comment), this does not seem to be working as quite a lot of results are NaN.

This was before my use of 0-11 cpu, should it change anything?

Not sure what's the problem but there was no analysis of the result.

If this is still happening after the change of 0-11 CPU I will revert it...

Currently the CI is locked

@thisalihassan

as it is still not working, reverted

@thisalihassan can you please take a look why all the NaN?

@rluvaton

@rluvaton sure also where can I see the logs?

You have this example: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1514/consoleFull

but you can run locally

FWIW there's a difference in output, e.g.
https://ci.nodejs.org/job/benchmark-node-micro-benchmarks/1510/console

21:42:04 "new","test_runner/suite-tests.js","concurrency='no' testType='async' testsPerSuite=1000 numberOfSuites=100",34812.26643390852,2.872550691

vs
https://ci.nodejs.org/job/benchmark-node-micro-benchmarks/1513/console

13:39:14 test_runner/suite-tests.js concurrency="no" testType="async" testsPerSuite=1000 numberOfSuites=100: 29,112.48474928658

i.e. as if the "new", (or "old",) field and opening " of the second field is missing.

also found this:

@rluvaton

@rluvaton nodejs/node#52456 "ipc" I untintentially removed this in the spawn stdio, that's why child.on message wasn't listening events

in Fork IPC is established by default but in spawn it is not established by default hence listener wasn't working

Navigation Menu

Repository navigation

Hetzner Benchmarking Machine Replacements #3657

28 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Hetzner Benchmarking Machine Replacements #3657

Description

Activity

targos commented on Mar 21, 2024

mcollina commented on Mar 21, 2024

ryanaslett commented on Mar 26, 2024

richardlau commented on Mar 26, 2024

richardlau commented on Mar 26, 2024

ryanaslett commented on Mar 26, 2024

richardlau commented on Mar 26, 2024

richardlau commented on Mar 26, 2024

ryanaslett commented on Mar 26, 2024

mhdawson commented on Mar 26, 2024

mhdawson commented on Mar 26, 2024

mhdawson commented on Mar 26, 2024

mhdawson commented on Mar 26, 2024

mhdawson commented on Mar 26, 2024

ryanaslett commented on Mar 26, 2024

28 remaining items

richardlau commented on Apr 8, 2024

rluvaton commented on Apr 8, 2024

thisalihassan commented on Apr 8, 2024

rluvaton commented on Apr 8, 2024

targos commented on Apr 8, 2024

mcollina commented on Apr 9, 2024

rluvaton commented on Apr 9, 2024

mcollina commented on Apr 9, 2024

rluvaton commented on Apr 9, 2024

rluvaton commented on Apr 10, 2024

thisalihassan commented on Apr 10, 2024

rluvaton commented on Apr 10, 2024

richardlau commented on Apr 10, 2024

rluvaton commented on Apr 10, 2024

thisalihassan commented on Apr 10, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions