concurrency fails w/o warmup #695

Activity

1. bjarthur

   added

   bugSomething isn't workingSomething isn't working

   on Oct 15, 2025
2. bjarthur commented on Oct 15, 2025

   

   bjarthur

   on Oct 15, 2025

   Author

   More actions

   here's another similar example. without executing jl.mwejl2(10,10) first, the call to submit hangs. curiously, mwejl() does not need to be warmed up.

   Python 3.13.2 | packaged by conda-forge | (main, Feb 17 2025, 14:10:22) [GCC 13.3.0] on linux
   Type "help", "copyright", "credits" or "license" for more information.
   >>> from concurrent import futures
   >>> from juliacall import Main as jl
   >>> jl.seval("""
   ...     function mwejl()
   ...         [x^2*y for x=1:10, y=1:10]
   ...     end
   ...     function mwejl2(sx, sy)
   ...         [x^2*y for x=1:sx, y=1:sy]
   ...     end
   ...     """)
   ... 
   Julia: mwejl2 (generic function with 1 method)
   >>> def mwepy(z):
   ...   print('1')
   ...   d = jl.mwejl()
   ...   print('2')
   ...   d2 = jl.mwejl2(10, 10)
   ...   print('3')
   ...   return (d,d2)
   ... 
   >>> exc = futures.ThreadPoolExecutor(max_workers=4)
   >>> jl.mwejl2(10,10)       # the next line hangs w/o this line
   Julia:
   10×10 Matrix{Int64}:
      1    2    3    4    5    6    7    8    9    10
      4    8   12   16   20   24   28   32   36    40
      9   18   27   36   45   54   63   72   81    90
     16   32   48   64   80   96  112  128  144   160
     25   50   75  100  125  150  175  200  225   250
     36   72  108  144  180  216  252  288  324   360
     49   98  147  196  245  294  343  392  441   490
     64  128  192  256  320  384  448  512  576   640
     81  162  243  324  405  486  567  648  729   810
    100  200  300  400  500  600  700  800  900  1000
   >>> f = exc.submit(mwepy, 0)
   1
   2
    3
   >>> f.result()
   (Julia:
   10×10 Matrix{Int64}:
      1    2    3    4    5    6    7    8    9    10
      4    8   12   16   20   24   28   32   36    40
      9   18   27   36   45   54   63   72   81    90
     16   32   48   64   80   96  112  128  144   160
     25   50   75  100  125  150  175  200  225   250
     36   72  108  144  180  216  252  288  324   360
     49   98  147  196  245  294  343  392  441   490
     64  128  192  256  320  384  448  512  576   640
     81  162  243  324  405  486  567  648  729   810
    100  200  300  400  500  600  700  800  900  1000, Julia:
   10×10 Matrix{Int64}:
      1    2    3    4    5    6    7    8    9    10
      4    8   12   16   20   24   28   32   36    40
      9   18   27   36   45   54   63   72   81    90
     16   32   48   64   80   96  112  128  144   160
     25   50   75  100  125  150  175  200  225   250
     36   72  108  144  180  216  252  288  324   360
     49   98  147  196  245  294  343  392  441   490
     64  128  192  256  320  384  448  512  576   640
     81  162  243  324  405  486  567  648  729   810
    100  200  300  400  500  600  700  800  900  1000)
   >>> jl.VERSION
   Julia: v"1.12.0"
   >>> import juliacall
   >>> juliacall.__version__
   '0.9.28'
   
3. bjarthur commented on Oct 15, 2025

   

   bjarthur

   on Oct 15, 2025

   Author

   More actions

   i'm guessing the compilation is allocating, as disabling GC with jl.GC.enable(False) obviates the need for warmup.
4. cjdoris commented on Oct 16, 2025

   

   cjdoris

   on Oct 16, 2025

   Member

   More actions

   AFAIU any Julia code that yields will hang when run from a Python thread other than the main one. I think it's because Julia doesn't know about the thread so the task scheduler never returns there.

   Presumably this is the same issue - compilation probably yields.

   There's a C function to let Julia adopt a foreign thread but I didn't have any luck using it the one time I tried.