Maniphest T76490

Included USD libraries cause symbol conflict with addons using USD binding
Confirmed, NormalTO DO
Actions

Assigned To
Sybren A. Stüvel (sybren)
Authored By
Brian Savery (bsavery)
May 7 2020, 4:03 AM
Tags
  • BF Blender
  • Pipeline, Assets & I/O
  • USD
Subscribers
Ankit Meel (ankitm)
Brian Savery (bsavery)
Jared Webber (lvxejay)
Jason schleifer (jasonschleifer)
JT Nelson (jta)
Karl Hendrikse (karlhendrikse)
Mel Massadian (melmass)
1 More Subscribers

Description

System Information
Operating system: macOS
Graphics card: N/A

Blender Version
Broken: 2.83 beta
Worked: Never

Included USD libraries cause symbol conflict with addons using USD python bindings. That is, if an addon wants to use USD in it (as a python c-extension most likely), there is an error:

"FATAL ERROR: [TF_DEBUG_ENVIRONMENT_SYMBOL] multiple symbol definitions.  This is usually due to software misconfiguration.  Contact the build team for assistance. (duplicate 'TF_SCRIPT_MODULE_LOADER')
in _Add at line 96 of /Users/brecht/dev/build_darwin/deps/build/usd/src/external_usd/pxr/base/lib/tf/debug.cpp"

Exact steps for others to reproduce the error

  1. Download and extract a build of USD. These ones have python3 bindings (I built my own). https://developer.nvidia.com/usd#bin
  2. Open python editor in blender.
  3. Run commands:
import sys
sys.path.append("PATH_TO_USD/lib/python")   # this is to get the pythonpath right
from pxr import Usd

So I can see two possible solutions to this.

  1. include python bindings with Blender
  2. somehow fix the symbol names in the blender side or in the external USD build so they don't clash. I tried to do this with specifying a different namespace name than "pxr" in the USD build but this didn't seem to help.

3, link python bindings of USD against blender.exe (Not sure how/if this would work)

Willing to work on the solution, but wanted input from Blender developers.

Related Objects

Event Timeline

Brian Savery (bsavery) created this task.May 7 2020, 4:03 AM
Brian Savery (bsavery) updated the task description.
Brian Savery (bsavery) updated the task description.May 7 2020, 4:19 AM
Sybren A. Stüvel (sybren) added a project: Pipeline, Assets & I/O.May 7 2020, 9:49 AM
Sybren A. Stüvel (sybren) closed this task as Archived.May 7 2020, 10:02 AM
Sybren A. Stüvel (sybren) claimed this task.
Sybren A. Stüvel (sybren) added a subscriber: Sybren A. Stüvel (sybren).

Hi Brian!

Broken: 2.83 beta

For future reference, unless a version is an official release, please include the Git hash in reports. The easiest to get this from Blender itself is to click Help → Report a Bug.

include python bindings with Blender

For this the USD library needs to support Python 3.7 first. It's still only compatible with Python 2.7. I don't think Blender should switch to a third party version of the library either.

somehow fix the symbol names in the blender side or in the external USD build so they don't clash. I tried to do this with specifying a different namespace name than "pxr" in the USD build but this didn't seem to help.

This is done in Blender as well (see -DPXR_SET_INTERNAL_NAMESPACE=usdBlender in usd.cmake), so the USD library should not conflict. If this namespace is not used in certain parts of the USD library, that would defeat the purpose of that option. This would be a bug in USD, though, and not in Blender.

Brian Savery (bsavery) changed the task status from Archived to Resolved.May 7 2020, 2:58 PM

@Sybren A. Stüvel (sybren)

You're mistaken about the python3 compatibility:
https://github.com/PixarAnimationStudios/USD/blob/master/CHANGELOG.md#build
USD 20.05 supports Python 3:
"Added support for Python 3. This may be enabled by specifying PXR_USE_PYTHON_3=TRUE when running cmake or by running build_usd.py with Python 3."

The nvidia ones have 3.6 support but you can build master with 3.7: https://developer.nvidia.com/usd-20.02-binary-windows-vs2017-python-3.6

Regardless, if I was using a USD builds with python 2 bindings I would not have gotten to this point.

Brian Savery (bsavery) added a comment.May 7 2020, 3:02 PM

Furthermore. After filing this I am able to work around this by building blender from source without USD support (which obviously removes the symbol conflict), but I would claim that's not a valid workaround for most users

Brian Savery (bsavery) changed the task status from Resolved to Archived.May 7 2020, 3:04 PM

Not sure about status how that got changed. Sybren marked it in valid, which I think is erroneous.

Brian Savery (bsavery) added a comment.May 8 2020, 4:13 AM

Might this be related to https://developer.blender.org/T76442 ?

Sybren A. Stüvel (sybren) reopened this task as Needs Information from User.May 8 2020, 10:39 AM
In T76490#926093, @Brian Savery (bsavery) wrote:

You're mistaken about the python3 compatibility:
https://github.com/PixarAnimationStudios/USD/blob/master/CHANGELOG.md#build
USD 20.05 supports Python 3:

I'm very happy to be wrong here :) Still, on Linux they only test with Python 3.6, whereas both the VFX Reference Platform and Blender are on 3.7. That's an infintesemally small difference compared to 2.7 though!

If it is indeed related to T76442, @Brian Savery (bsavery) could you maybe test what happens when you build Blender with the conflicting symbols added to source/creator/blender.map? That should hide those symbols; if that is enough I'd be happy to include that in Blender and get things moving forward.

Brian Savery (bsavery) added a comment.May 8 2020, 3:34 PM

Yep will test. Here's what I've done so far:

  1. macOS with standard blender build - Error above
  2. macOS build with USD disabled - Ok
  3. ubuntu build with 3.7 build of USD - Ok.
Bastien Montagne (mont29) changed the subtype of this task from "Report" to "To Do".May 12 2020, 9:58 AM
Bastien Montagne (mont29) moved this task from Backlog to TODO's on the Pipeline, Assets & I/O board.
JT Nelson (jta) added a subscriber: JT Nelson (jta).May 15 2020, 6:45 AM
Ankit Meel (ankitm) changed the task status from Needs Information from User to Confirmed.May 22 2020, 4:55 PM
Ankit Meel (ankitm) added a subscriber: Ankit Meel (ankitm).

(since it's a todo)

Bastien Montagne (mont29) moved this task from TODO's to Long-Term - Official on the Pipeline, Assets & I/O board.Jun 11 2020, 6:43 PM
Jared Webber (lvxejay) added a subscriber: Jared Webber (lvxejay).Jun 26 2020, 6:00 PM
Mel Massadian (melmass) added a subscriber: Mel Massadian (melmass).Dec 10 2020, 12:11 AM
Yegor (Yegor) added a subscriber: Yegor (Yegor).Dec 10 2020, 12:34 AM
Mel Massadian (melmass) added a comment.EditedAug 9 2021, 6:36 PM

Is there any advancement on this issue?
I think that adding a custom namespace is the solution, like specified here:

https://github.com/PixarAnimationStudios/USD/blob/release/BUILDING.md#usd-developer-options

Mel Massadian (melmass) added a comment.Aug 23 2021, 5:14 AM

This is also an issue for projects like https://github.com/GPUOpen-LibrariesAndSDKs/BlenderUSDHydraAddon

But apparently only on macOS?

Jason schleifer (jasonschleifer) added a subscriber: Jason schleifer (jasonschleifer).Sep 8 2021, 3:31 AM
Sybren A. Stüvel (sybren) added a comment.EditedJan 31 2022, 11:28 AM

Blender already builds USD with a custom namespace: https://developer.blender.org/diffusion/B/browse/master/build_files/build_environment/cmake/usd.cmake$36

I'm certainly open to suggestions on how to solve this issue.

Bastien Montagne (mont29) added a project: USD.Mar 15 2022, 5:38 PM
Karl Hendrikse (karlhendrikse) added a subscriber: Karl Hendrikse (karlhendrikse).EditedJul 11 2022, 6:51 AM

Using a custom namespace doesn't fix this because the problem isn't a symbol collision.

The problem is that when any shared library is loaded into the process, the USD library looks in the library's Mach-O headers and pulls out function pointers for all special "constructors" and calls them. The USD library contains a macro one can use to declare such "constructors", and they are stored in ".pxrctor" sections in the binary. Some of the constructors declared in the USD library cause the application to abort when run multiple times, which is what happens when there are multiple instances of the library loaded, both searching for ".pxrctor" sections and calling all "constructors" when libraries are loaded.

One solution I found is to change the name used for the sections in one instance of the USD library so the instances aren't trying to run each other's "constructors". Changing all occurrences of pxrctor into fooctor (and likewise pxrdtor into foodtor) in the USD library fixed the problem for me.

(I had this problem in a separate application, but I found this page when searching for the solution and after figuring it out for myself I thought posting here might be helpful.)