Skip to content

test-shadow-realm-gc-module can crash on macOS (OOM) #50726

Closed
Closed
test-shadow-realm-gc-module can crash on macOS (OOM)#50726
Labels
macosIssues and PRs related to the macOS platform.testIssues and PRs related to Node.js core tests and test infrastructure.

Description

@targos
targos
opened on Nov 14, 2023
$ out/Debug/node --experimental-shadow-realm --max-old-space-size=20 /Users/mzasso/git/nodejs/node/test/parallel/test-shadow-realm-gc-module.js

<--- Last few GCs --->

[61276:0x148008000]     3114 ms: Scavenge 17.0 (20.1) -> 16.4 (25.1) MB, 1.33 / 0.00 ms  (average mu = 0.889, current mu = 0.757) allocation failure; 
[61276:0x148008000]     3147 ms: Mark-Compact 20.8 (25.8) -> 18.0 (28.6) MB, 11.12 / 0.00 ms  (+ 0.7 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 39 ms) (average mu = 0.867, current mu = 0.858) allocation fail

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x1026d59c8 node::DumpNativeBacktrace(__sFILE*) [/Users/mzasso/git/nodejs/node/out/Debug/node]
 2: 0x102814ef4 node::Abort() [/Users/mzasso/git/nodejs/node/out/Debug/node]
 3: 0x1028151d8 node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [/Users/mzasso/git/nodejs/node/out/Debug/node]
 4: 0x102b9f8f0 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/mzasso/git/nodejs/node/out/Debug/node]
 5: 0x1030362f8 v8::internal::Heap::stack() [/Users/mzasso/git/nodejs/node/out/Debug/node]
 6: 0x1030341d4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/mzasso/git/nodejs/node/out/Debug/node]
...
79: 0x1042501c4 main [/Users/mzasso/git/nodejs/node/out/Debug/node]
80: 0x18157d0e0 start [/usr/lib/dyld]
[1]    61276 abort      out/Debug/node --experimental-shadow-realm --max-old-space-size=20

Activity

  1. targos
    added
    testIssues and PRs related to Node.js core tests and test infrastructure.
    macosIssues and PRs related to the macOS platform.
    on Nov 14, 2023

  2. MrJithil commented on Nov 14, 2023

    @MrJithil
    MrJithil
    on Nov 14, 2023
    Member
    node:internal/process/task_queues:95
    runMicrotasks();
    ^

TypeError: Cannot import in ShadowRealm (TypeError: A dynamic import callback was not specified.)

    I ran on macOS locally, above is the error message I got with Release/node

  3. targos
    changed the title [-]`test-shadow-realm-gc-module` crashes on macOS in debug build[/-] [+]`test-shadow-realm-gc-module` can crash on macOS (OOM)[/+] on Nov 14, 2023

  4. targos commented on Nov 14, 2023

    @targos
    targos
    on Nov 14, 2023
    MemberAuthor

    I just tried with the release build and I get the same OOM error. I updated the title.

  5. targos commented on Nov 14, 2023

    @targos
    targos
    on Nov 14, 2023
    MemberAuthor

    /cc @legendecas who added the test in #48655

  6. legendecas commented on Nov 14, 2023

    @legendecas
    legendecas
    on Nov 14, 2023
    Member

    I checked the last two GitHub action runs on the main branch and I didn't find this failure on the macOS build (https://github.com/nodejs/node/actions/runs/6859065494/job/18650800024 failed for a different test). And I didn't reproduce the failure with 1000 runs locally. Does this problem constantly reproduce with your local build?

    Stress test: https://ci.nodejs.org/job/node-stress-single-test/464

  7. targos commented on Nov 14, 2023

    @targos
    targos
    on Nov 14, 2023
    MemberAuthor

    I am not close to my computer now but it reproduced consistently (I tried to run it between 5 and 10 times )

  8. MrJithil commented on Nov 14, 2023

    @MrJithil
    MrJithil
    on Nov 14, 2023
    Member

    With the latest merge? I took the latest and now unable to reproduce

    for i in {1..100}; do ./node --experimental-shadow-realm --max-old-space-size=20 test/parallel/test-shadow-realm-gc-module.js; done

  9. targos commented on Nov 14, 2023

    @targos
    targos
    on Nov 14, 2023
    MemberAuthor

    I have a arm64 Mac. Maybe it's platform specific

  10. MrJithil commented on Nov 14, 2023

    @MrJithil
    MrJithil
    on Nov 14, 2023
    Member

    I am using M1 Pro chip. Is there any customisations you did like node max heap size?

  11. targos commented on Nov 14, 2023

    @targos
    targos
    on Nov 14, 2023
    MemberAuthor

    tools/test.py parallel/test-shadow-realm-gc-module --repeat 100
    [00:10|% 100|+ 1|- 99]: Done

    I am using M1 Pro chip. Is there any customisations you did like node max heap size?

    I always build with ./configure --ninja --node-builtin-modules-path=$(pwd) in case that's relevant.

  12. MrJithil commented on Nov 14, 2023

    @MrJithil
    MrJithil
    on Nov 14, 2023
    Member

    Let me check.

  13. marco-ippolito commented on Nov 14, 2023

    @marco-ippolito
    marco-ippolito
    on Nov 14, 2023
    Member

    I've seen this failure in CI on different platforms, https://ci.nodejs.org/job/node-test-commit-linuxone/nodes=rhel8-s390x/40888/console

  14. legendecas commented on Nov 14, 2023

    @legendecas
    legendecas
    on Nov 14, 2023
    Member

    I can confirm this can be reproduced with the configure flag --node-builtin-modules-path, in either the release build or the debug build.

  15. joyeecheung commented on Nov 14, 2023

    @joyeecheung
    joyeecheung
    on Nov 14, 2023
    Member

    I think it's probably caused by the limit (20MB) being a bit too small for --node-builtin-modules-path which reads the JS sources from disks and creates new strings for every realm, instead of what we normally do (using externalized strings shared across the realms). And then the test is running the allocation in a hot loop so the GC cannot keep up. Awaiting a promisified setTimeout in the loop to give the GC some breathing room fixes it locally for me.

  16. joyeecheung commented on Nov 17, 2023

    @joyeecheung
    joyeecheung
    on Nov 17, 2023
    Member

    I am now seeing test-shadow-realm-gc flakes in V8's Node.js integration test CI too (in different unrelated CLs). We should at least try to get #50735 landed quickly. If that's still not enough to fix it we should mark it so to avoid spreading this flake to V8.

  17. joyeecheung
    added a commit that references this issue on Nov 17, 2023

    test: give more time to GC in test-shadow-realm-gc-*

    c5eb2c4
  18. targos
    added a commit that references this issue on Nov 23, 2023

    test: give more time to GC in test-shadow-realm-gc-*

    91dbe13
  19. lucshi
    added a commit that references this issue on Nov 27, 2023

    test: give more time to GC in test-shadow-realm-gc-*

    377c28d

  20. targos commented on Nov 28, 2023

    @targos
    targos
    on Nov 28, 2023
    MemberAuthor

    Can't reproduce anymore. We can reopen if it happens again.

  21. targos
    closed this as completedon Nov 28, 2023
  22. RafaelGSS
    added 2 commits that reference this issue on Nov 29, 2023

    test: give more time to GC in test-shadow-realm-gc-*

    6475cee
  23. richardlau
    added a commit that references this issue on Mar 25, 2024

    test: give more time to GC in test-shadow-realm-gc-*

    26a06b0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    macosIssues and PRs related to the macOS platform.testIssues and PRs related to Node.js core tests and test infrastructure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions