V8 gyp scaffolding responsibility handoff

@targos

Refs: nodejs/CTC#76 (comment)
Refs: nodejs/CTC#2 (comment)

... the legacy GYP config will be removed from V8’s repository, but stays in node/deps/v8. Keeping it in sync with GN build config becomes Node.js’ responsibility ...

AFAICT V8 has already stopped CI testing the .gyp scaffolding, and are planning to remove the .gyp files by the end of the year.

Should we designate an "owner" for those files?
Should we keep upstreaming fixes meanwhile (deps: update V8 to 6.2.414.44 node#16848 (comment))?

/CC @nodejs/v8 @nodejs/gyp @nodejs/build @targos @hashseed

Our bot is still running the regular Node make with gyp just like official Node.

Precisely. We still test Node with gyp. We have plans to change this, and will notify once we have something usable.

Our bot is still running the regular Node make with gyp just like official Node.

Obviously any CI coverage provided by V8 is greatly appreciated!
AFAICT there seems to be coverage missing - ARM was broken for a while, and Windows wasn't covered (maybe just building with MSBuild vs. ninja? While we use MSBuild and both VS2015 and VS2017).

IMHO the deprecation of the gyp scaffolding, and the pending removal is starting to be felt by node, and I'm just saying we should be ready. This way we won't need to scramble last minute to smooth over build problems when we want to backport fixes (nodejs/node#16848 (comment)).

Another point is that ATM reviewing these patches is cumbersome, since they can only land together with V8 backports or version bumps, see nodejs/node#16848 (comment)

To reiterate, we already have knowledge and time investment in maintaining GYP (the tool), so taking ownership of the tool and the .gyp files is not too far fetched.

@hashseed

@hashseed It's 2018, any news?

In the meantime I created a wrapper script as planned to create a libv8_monolith.a static library that can be used to link node instead of libv8_base.a. I haven't created a PR on the Node.js side yet to make things work smoothly.

To build node without using V8's gyp configs, perform following steps:

check out branch vee-eight-lkgr from https://github.com/v8/node
apply following patch

diff --git a/common.gypi b/common.gypi
index ba44178169..2e16bacb2b 100644
--- a/common.gypi
+++ b/common.gypi
@@ -48,7 +48,7 @@
         'V8_BASE': '<(PRODUCT_DIR)/obj/deps/v8/src/libv8_base.a',
        }, {
          'OBJ_DIR%': '<(PRODUCT_DIR)/obj.target',
-         'V8_BASE%': '<(PRODUCT_DIR)/obj.target/deps/v8/src/libv8_base.a',
+         'V8_BASE%': '<(PRODUCT_DIR)/obj.target/v8_monolith/geni/gn/obj/libv8_monolith.a',
       }],
       ['OS == "win"', {
         'os_posix': 0,
diff --git a/node.gypi b/node.gypi
index 718ef0c9e7..e88bc0520b 100644
--- a/node.gypi
+++ b/node.gypi
@@ -17,7 +17,7 @@
     }],
     [ 'node_use_bundled_v8=="true"', {
       'dependencies': [
-        'deps/v8/src/v8.gyp:v8',
+        'deps/v8/src/v8.gyp:v8_monolith',
         'deps/v8/src/v8.gyp:v8_libplatform'
       ],
     }],

chdir into the node checkout
run deps/v8/tools/node/fetch_deps.py deps/v8 force to fetch the necessary dependencies.
run ./configure && make -j 4 node. This uses deps/v8/tools/node/build_gn.py to build out/Release/obj.target/deps/v8/src/libv8_monolith.a to link into the final executable.

I only tested this with a static build for x64 linux. Once that's usable and streamlined I'll work on a few more build configurations.

Is there any difference between v8 and v8_monolith other than the way it's built?

Not really. Except that v8 is a collection of files and v8_monolith just one.

run deps/v8/tools/node/fetch_deps.py deps/v8 to fetch the necessary dependencies.

The file deps/v8/tools/node/fetch_deps.py does not exist.

Did you check out from github.com/v8/node? I landed it in the last few weeks so it's not yet in node master.

Yes, there's no node directory here: https://github.com/v8/node/tree/vee-eight-lkgr/deps/v8/tools

You are right. Turns out anything called node is ignored by git due to Node's root .gitignore. So when I updated our fork's V8 via script, the whole folder was not added.

I fixed that in our node fork here. I'll upstream that soonish. Thanks for making me aware!

In the meantime I also push a new version of V8 to our fork, with the node folder.

fetch_deps.py doesn't do anything. It seems intentional: https://github.com/v8/node/blob/bc4ca529a007c9f23acd87534197086b2f10e65e/deps/v8/tools/node/fetch_deps.py#L89-L91

Seems to work now.

Is this the final solution? I'm curious why you went with a fetch_deps.py script. I understand that checking in binaries is not ideal, but relying on a remote resource makes it susceptible to link rot, consequently there's no guarantee that this commit will be buildable in the future.

Thanks for verifying!

This is not the "final" solution in the sense of that's where Node.js' build system should end up. The problem this should solve is that the V8 team is committed to test against Node.js, but wants to end support for gyp. This is not designed as a full replacement for V8's gyp configs.

In the short term, Node.js inherits ownership over V8's gyp configurations to continue building with pure gyp configs (no GN wrapper). Ultimately Node.js will have to figure out whether to keep maintaining its own and V8's gyp configurations, or find something better altogether.

These remote resources are ones that V8 relies on and are tested on our infrastructure, so for the purpose of V8 testing against Node.js, link rot is not an issue. However, for building Node.js without having to rely on V8 and Chrome resources, this would indeed be a problem, so the gyp configs have to be maintained.

Ultimately Node.js will have to figure out whether to keep maintaining its own and V8's gyp configurations, or find something better altogether.

That's the issue - if Node.js were to consider adopting GN (at least for V8), the dependence on V8 and Chrome resources might become an obstacle, considering the Node.js repo is currently self-contained.

Having talked to the GN team in the past, I don't think GN is a viable option for Node.js to completely follow through. This gyp-gn bridge is really just to cut the dependency to gyp for our integration testing.

Having talked to the GN team in the past, I don't think GN is a viable option for Node.js to completely follow through.

Any reason for that besides the dependence on remote resources? And could you clarify what PR you meant in #411 (comment)? I assumed it referred to the GN wrapper.

The doc lays out a couple of reasons. Mostly because GN does not support the full range of platforms that Node.js does.

The PR I have in mind is to introduce a build flag to Node.js to be able to use the gn wrapper, currently the patch in step 2.

@refack

Mostly because GN does not support the full range of platforms that Node.js does.

According to @refack, the V8 support for those platforms that are not supported by GN is maintained by companies interested in such support (IRC log). So it seems reasonable to expect those companies to help with GN support on those platforms as well if Node.js were to follow this route. It seems to me that it would be more efficient in the long run than maintaining gyp configs.

@mhdawson

A couple of thoughts:

Platforms not backed by companies, such as FreeBSD, will not receive any support.
I have been wanting to asking @mhdawson for ever how the GN support for S390 and PPC looks. I heard rumors that there was something being worked on.
GN relies on a number of platform-dependent configs and binaries. You will not get around of fetching these resources from somewhere, unless you want to include them in Node altogether.
GN is not designed for broad public adoption from what I gathered.
The advantage of GN over something new, e.g. cmake, is that V8 already comes with GN files. Configs for cmake would have to be maintained, just like for gyp in the near future. Bazel has the same advantage as GN.
I'm not even sure Chromium/V8 will stick with GN in, say, 5 years :P

Platforms not backed by companies, such as FreeBSD, will not receive any support.

Are there any other such platforms? FreeBSD is free and works on x86-64, so pretty much any collaborator can get involved with its support.

GN relies on a number of platform-dependent configs and binaries. You will not get around of fetching these resources from somewhere, unless you want to include them in Node altogether.

I think including them in Node is an option.

Bazel has the same advantage as GN.

In that case, Bazel is definitely worth looking into.

I'm not even sure Chromium/V8 will stick with GN in, say, 5 years :P

I'd say if the effort to adopt GN remains useful for 5 years, it would be more than worth it.

@hashseed

@hashseed I'll ask the people who were working on the GN support to comment here.

I streamlined things a bit.

git clone https://github.com/v8/node v8-node
cd v8-node 
git checkout vee-eight-lkgr
./configure --build-v8-with-gn
make -j4 test

Is there a next step here? Or can this issue be closed?

Guess next steps will have to happen when upgrading to V8 6.6.

Closing via nodejs/node#19201, feel free to re-open if needed.

Navigation Menu

Repository navigation

V8 gyp scaffolding responsibility handoff #411

7 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

V8 gyp scaffolding responsibility handoff #411

Description

Activity

fhinkel commented on Nov 7, 2017

hashseed commented on Nov 7, 2017

refack commented on Nov 7, 2017

refack commented on Dec 6, 2017

seishun commented on Jan 5, 2018

hashseed commented on Jan 5, 2018

seishun commented on Jan 5, 2018

hashseed commented on Jan 5, 2018

seishun commented on Jan 5, 2018

hashseed commented on Jan 5, 2018

seishun commented on Jan 5, 2018

hashseed commented on Jan 5, 2018

seishun commented on Jan 5, 2018

7 remaining items

seishun commented on Jan 6, 2018

hashseed commented on Jan 6, 2018

seishun commented on Jan 7, 2018

hashseed commented on Jan 7, 2018

seishun commented on Jan 7, 2018

hashseed commented on Jan 7, 2018

seishun commented on Jan 7, 2018

hashseed commented on Jan 7, 2018

seishun commented on Jan 7, 2018

mhdawson commented on Jan 8, 2018

hashseed commented on Jan 12, 2018

jasnell commented on Feb 17, 2018

hashseed commented on Feb 17, 2018

fhinkel commented on Mar 8, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions