Maniphest T76414

Blender crashing when trying to use Cupy
Closed, Resolved
Actions

Assigned To
Brecht Van Lommel (brecht)
Authored By
Nathan Kundtz (nkundtz)
May 4 2020, 7:35 PM
Tags
  • BF Blender
  • Add-ons (BF-Blender)
Subscribers
Ankit Meel (ankitm)
Brecht Van Lommel (brecht)
Jared Webber (lvxejay)
Nathan Kundtz (nkundtz)

Description

System Information
Operating system: Linux-5.3.0-51-generic-x86_64-with-debian-buster-sid 64 Bits
Graphics card: GeForce GTX 1050/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 440.59
CUDA: 10.2

Blender Version
Broken: version: 2.83 (sub 13), branch: master, commit date: 2020-04-13 08:07, hash: rB2ec6eca518ef
Broken: version: 2.83 (sub 15), branch: master, commit date: 2020-05-03 18:42, hash: rB331bf04fad93
Worked: version: 2.83 (sub 11), branch: master, commit date: 2020-03-31 23:07, hash: rBda3cb514e52d

Short description of error
Blender crashes (Segmentation Fault) while trying to use cupy.
It will pass the first three flags and then segfault before the fourth.

CudaTest.py551 BDownload

Exact steps for others to reproduce the error
From the command line use blender --background --python CudaTest.py
This will throw a seg fault
If you run the same script outside of blender:
$(BLENDER_DIR)/2.83/python/bin/python3.7m CudaTest.py
The code will run normally.

Environment setup
Assuming CUDA is installed normally and from a fresh blender install:
$(BLENDER_DIR)/2.83/python/bin/python3.7m -m ensurepip
$(BLENDER_DIR)/2.83/python/bin/python3.7m -m pip install cupy-cuda102

Much appreciation
Really appreciate your help! Thanks and hope everyone is staying safe!

Other information
Example backtrace of the core dump below.

#0  0x000000000d555b20 in nvrtcVersion ()
#1  0x00007fca78ca297d in pyx_f_4cupy_4cuda_5nvrtc_getVersion (pyx_skip_dispatch=<optimized out>)
    at cupy/cuda/nvrtc.cpp:1923
#2  0x00007fca78ca2afb in pyx_pf_4cupy_4cuda_5nvrtc_2getVersion (pyx_self=<optimized out>)
    at cupy/cuda/nvrtc.cpp:2021
#3  pyx_pw_4cupy_4cuda_5nvrtc_3getVersion (pyx_self=<optimized out>, unused=<optimized out>)
    at cupy/cuda/nvrtc.cpp:2008
#4  0x0000000005700a20 in _PyMethodDef_RawFastCallKeywords ()
#5  0x0000000005700ab5 in _PyCFunction_FastCallKeywords ()
#6  0x0000000000c0d098 in _PyEval_EvalFrameDefault ()
#7  0x0000000000c04b80 in ?? ()
#8  0x0000000000c0e8fe in _PyEval_EvalFrameDefault ()
#9  0x0000000000c04b80 in ?? ()
#10 0x0000000000c0e8fe in _PyEval_EvalFrameDefault ()
#11 0x00000000057b244c in _PyEval_EvalCodeWithName ()
#12 0x0000000005700536 in _PyFunction_FastCallKeywords ()
#13 0x0000000000c0e8fe in _PyEval_EvalFrameDefault ()
#14 0x00000000057b244c in _PyEval_EvalCodeWithName ()
#15 0x0000000005700380 in _PyFunction_FastCallDict ()
#16 0x00007fcaa487fe57 in __Pyx_PyObject_Call (kw=0x7fca5234b0a0, arg=0x7fca523e7980, func=<optimized out>)
    at cupy/core/core.cpp:53653
#17 pyx_f_4cupy_4core_4core_compile_with_cache (pyx_v_source=0x7fca5247e1d0, 
    pyx_skip_dispatch=<optimized out>, pyx_optional_args=<optimized out>) at cupy/core/core.cpp:35823
#18 0x00007fca703887a1 in pyx_f_4cupy_4core_7_kernelget_simple_elementwise_kernel (
    pyx_v_params=0x7fca527131e0, pyx_v_arginfos=0x7fca5234b910, pyx_v_operation=0x7fca5233d2d0, 
    pyx_v_name=0x7fca5277d490, pyx_v_type_map=<optimized out>, pyx_v_preamble=0x7fcae7c0e3b0, 
    pyx_skip_dispatch=0, pyx_optional_args=0x7ffdf824d010) at cupy/core/_kernel.cpp:4481
#19 0x00007fca70389127 in pyx_pf_4cupy_4core_7_kernel_2_get_simple_elementwise_kernel (
    pyx_v_options=<optimized out>, pyx_v_after_loop=<optimized out>, pyx_v_loop_prep=0x7fcae7c0e3b0, 
    pyx_v_preamble=<optimized out>, pyx_v_type_map=<optimized out>, pyx_v_name=0x7fca5277d490, 
    pyx_v_operation=0x7fca5233d2d0, pyx_v_arginfos=0x7fca5234b910, pyx_v_params=0x7fca527131e0, 
    pyx_self=<optimized out>) at cupy/core/_kernel.cpp:4711
#20 pyx_pw_4cupy_4core_7_kernel_3_get_simple_elementwise_kernel (pyx_self=<optimized out>, 
    pyx_args=<optimized out>, pyx_kwds=<optimized out>) at cupy/core/_kernel.cpp:4681
#21 0x0000000005702705 in PyCFunction_Call ()
#22 0x00007fca703860ff in Pyx_PyObject_Call (kw=0x7fca52348500, arg=0x7fca523e78a0, func=0x7fca79748500)
    at cupy/core/_kernel.cpp:36022
#23 pyx_pf_4cupy_4core_7_kernel_12_get_elementwise_kernel (pyx_v_kwargs=0x7fca52348500, 
    pyx_v_preamble=0x7fcae7c0e3b0, pyx_v_name=0x7fca5277d490, pyx_v_operation=0x7fca5233d2d0, 
    pyx_v_params=0x7fca527131e0, pyx_v_type_map=0x7fcaa4af4750, pyx_v_arginfos=0x7fca5234b910, 
    pyx_self=<optimized out>) at cupy/core/_kernel.cpp:15617
#24 pyx_pw_4cupy_4core_7_kernel_13_get_elementwise_kernel (__pyx_self=<optimized out>,
pyx_args=<optimized out>, pyx_kwds=<optimized out>) at cupy/core/_kernel.cpp:15281
#25 0x0000000005702705 in PyCFunction_Call ()
#26 0x00007fca862468a7 in Pyx_PyObject_Call (kw=0x7fca52344eb0, arg=0x7fca523e77c0, func=0x7fca7975a410)
    at cupy/util.cpp:4384
#27 pyx_pf_4cupy_4util_7memoize_9decorator_ret (pyx_v_kwargs=0x7fca52344eb0, pyx_v_args=0x7fca523e77c0, 
    pyx_self=<optimized out>) at cupy/util.cpp:2016
#28 pyx_pw_4cupy_4util_7memoize_9decorator_1ret (pyx_self=<optimized out>, pyx_args=0x7fca523e77c0, 
    pyx_kwds=<optimized out>) at cupy/util.cpp:1831
#29 0x00007fca703587f5 in Pyx_PyObject_Call (func=0x7fca7975e050, arg=0x7fca523e77c0, kw=0x7fca52709730)
    at cupy/core/_kernel.cpp:36022
#30 0x00007fca70361fa3 in pyx_f_4cupy_4core_7_kernel_17ElementwiseKernelget_elementwise_kernel (
    pyx_v_self=0x7fca52711d60, pyx_v_dev_id=<optimized out>, pyx_v_arginfos=0x7fca5234b910, 
    pyx_v_type_map=0x7fcaa4af4750, pyx_skip_dispatch=<optimized out>) at cupy/core/_kernel.cpp:17938
#31 0x00007fca70379f9f in pyx_pf_4cupy_4core_7_kernel_17ElementwiseKernel_2call (
    pyx_v_kwargs=0x7fcabe957690, pyx_v_args=0x7fcabe953b50, pyx_v_self=0x7fca52711d60)
    at cupy/core/_kernel.cpp:17297
#32 pyx_pw_4cupy_4core_7_kernel_17ElementwiseKernel_3call (pyx_v_self=0x7fca52711d60, 
    pyx_args=0x7fcabe953b50, __pyx_kwds=<optimized out>) at cupy/core/_kernel.cpp:16373
#33 0x0000000005700bd4 in _PyObject_FastCallKeywords ()
#34 0x0000000000c0ca72 in _PyEval_EvalFrameDefault ()
#35 0x00000000057b244c in _PyEval_EvalCodeWithName ()
pyx_v_type_map=0x7fcaa4af4750, pyx_skip_dispatch=<optimized out>) at cupy/core/_kernel.cpp:17938
#31 0x00007fca70379f9f in pyx_pf_4cupy_4core_7_kernel_17ElementwiseKernel_2call (
    pyx_v_kwargs=0x7fcabe957690, pyx_v_args=0x7fcabe953b50, pyx_v_self=0x7fca52711d60)
    at cupy/core/_kernel.cpp:17297
#32 pyx_pw_4cupy_4core_7_kernel_17ElementwiseKernel_3call (pyx_v_self=0x7fca52711d60, 
    pyx_args=0x7fcabe953b50, pyx_kwds=<optimized out>) at cupy/core/_kernel.cpp:16373
#33 0x0000000005700bd4 in _PyObject_FastCallKeywords ()
#34 0x0000000000c0ca72 in _PyEval_EvalFrameDefault ()
#35 0x00000000057b244c in _PyEval_EvalCodeWithName ()
#36 0x0000000005700536 in _PyFunction_FastCallKeywords ()
#37 0x0000000000c0dd52 in _PyEval_EvalFrameDefault ()
#38 0x00000000057b244c in _PyEval_EvalCodeWithName ()
#39 0x0000000005700536 in _PyFunction_FastCallKeywords ()
#40 0x0000000000c0e50c in _PyEval_EvalFrameDefault ()
#41 0x00000000057b244c in _PyEval_EvalCodeWithName ()
#42 0x00000000057b257e in PyEval_EvalCodeEx ()
#43 0x00000000057b25ab in PyEval_EvalCode ()
#44 0x00000000057e57a2 in PyRun_FileExFlags ()
#45 0x00000000011ec931 in ?? ()
#46 0x0000000000c10ab8 in _start ()

Revisions and Commits

rB Blender
rB9d8a5834829c Fix T76414: crash using Python module that uses NVRTC

Related Objects

Mentioned In
T76442: Hide Blender symbols by default on all platforms
Mentioned Here
rBb555b8dedce0: Build: hide most symbols on macOS on Linux to avoid conflicts
rBf1573731bc8b: UI: Support split property layout for pointer search buttons
rBbae1c243cec8: Build: hide USD symbols, make Blender symbols visible again

Event Timeline

Nathan Kundtz (nkundtz) created this task.May 4 2020, 7:35 PM
Jared Webber (lvxejay) added a subscriber: Jared Webber (lvxejay).May 4 2020, 7:46 PM
Ankit Meel (ankitm) updated the task description.May 4 2020, 9:18 PM
Ankit Meel (ankitm) added a subscriber: Ankit Meel (ankitm).May 5 2020, 8:34 AM
Philipp Oeser (lichtwerk) added a project: Add-ons (BF-Blender).May 5 2020, 11:19 AM
Jared Webber (lvxejay) added a comment.May 5 2020, 11:34 AM

Looks like the bug might have something to do with the symbol visibility changes introduced in bae1c243cec: https://developer.blender.org/rBbae1c243cec809ac783c1b8f2fb65a570b526ef0

I tested the previous commit, f1573731bc8, which works as expected, no segfault: https://developer.blender.org/rBf1573731bc8b80c03ef5db4292568781969846c3

Brecht Van Lommel (brecht) changed the task status from Needs Triage to Needs Information from User.EditedMay 5 2020, 11:59 AM
Brecht Van Lommel (brecht) added a subscriber: Brecht Van Lommel (brecht).

Does this work in 2.82?

What I'm guessing is that this briefly worked after rBb555b8dedce0: Build: hide most symbols on macOS on Linux to avoid conflicts, then part of that got reverted in rBbae1c243cec8: Build: hide USD symbols, make Blender symbols visible again and it stopped working again?

Brecht Van Lommel (brecht) mentioned this in T76442: Hide Blender symbols by default on all platforms.May 5 2020, 12:22 PM
Nathan Kundtz (nkundtz) added a comment.May 5 2020, 3:44 PM

Based on my testing and that of the NVIDIA team Segmentation faults are produced in all of our tested 2.82, 2.79, or alpha 2.90 releases.

Brecht Van Lommel (brecht) closed this task as Resolved by committing rB9d8a5834829c: Fix T76414: crash using Python module that uses NVRTC.May 5 2020, 4:03 PM
Brecht Van Lommel (brecht) claimed this task.
Brecht Van Lommel (brecht) added a commit: rB9d8a5834829c: Fix T76414: crash using Python module that uses NVRTC.