Maniphest T76993

user site-packages path no longer in sys.path, resulting in modules installed there can no longer be imported any more by Blender's python
Closed, Archived
Actions

Assigned To
Campbell Barton (campbellbarton)
Authored By
MACHIN3 (MACHIN3)
May 23 2020, 1:52 PM
Tags
  • Python API
  • BF Blender
Subscribers
Brecht Van Lommel (brecht)
Campbell Barton (campbellbarton)
CansecoGPC (CansecoGPC)
James (martiaus1028)
MACHIN3 (MACHIN3)
Robert Guetzkow (rjg)
stephen leger (stephen_leger)

Description

System Information
Operating system: Linux-4.15.0-96-generic-x86_64-with-debian-buster-sid 64 Bits
Graphics card: GeForce GTX 1050/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 435.21

Blender Version
Broken: version: 2.83 (sub 15), branch: master, commit date: 2020-05-03 18:42, hash: rB331bf04fad93
Worked: 'rB1239cab11ff9'

Short description of error
Blender no longer considers python modules installed to the user site-packages path (if it exists), as of rB331bf04fad93(first build I've noticed it on, probably earlier though).
As a consequence any modules installed in this location can no longer be imported from Blender's python.

The user site-packages path on Windows is
C:\Users\name\AppData\Roaming\Python\Python37\site-packages
On Linux
/home/name/.local/lib/python3.7/site-packages
On MacOS (I think)
/Users/name/Library/Python/3.7/lib/python/site-packages

It can be fetched by doing

import site
site.getusersitepackages()

Here are two system_info files to demonstrate it:

system-info_31bf04fad93.txt21 KBDownload
user site-packages path missing
system-info_1239cab11ff9.txt21 KBDownload
user site-packages path present: /home/x/.local/lib/python3.7/site-packages

Exact steps for others to reproduce the error

  • save out the system info in rB1239cab11ff9 or earlier, as well as rB331bf04fad93 or newer
  • alternatively run the following snippet in the python console
  • look for the user site-packages path
import sys
for p in sys.path:
    print(p)

Related Objects

Mentioned Here
T71420: Using PyPI from Blender
rBS79a58eef059f: Fix T73977, T73825: ignore Python user site-packages directory by default
rB1239cab11ff9: Fix T75203: Crash when changing active keying set

Event Timeline

MACHIN3 (MACHIN3) renamed this task from user site-packages path no longer in sys.path, resulting in modules installed there can't be imported any more by Blender's python to user site-packages path no longer in sys.path, resulting in modules installed there can no longer be imported any more by Blender's python.May 23 2020, 1:52 PM
MACHIN3 (MACHIN3) created this task.
MACHIN3 (MACHIN3) updated the task description.
MACHIN3 (MACHIN3) updated the task description.
MACHIN3 (MACHIN3) updated the task description.May 23 2020, 1:58 PM
MACHIN3 (MACHIN3) updated the task description.
MACHIN3 (MACHIN3) updated the task description.May 23 2020, 2:03 PM
MACHIN3 (MACHIN3) updated the task description.
MACHIN3 (MACHIN3) updated the task description.
MACHIN3 (MACHIN3) updated the task description.
MACHIN3 (MACHIN3) updated the task description.May 23 2020, 2:07 PM
MACHIN3 (MACHIN3) added a comment.May 23 2020, 2:12 PM

Apparently, its intentional? https://developer.blender.org/rBS79a58eef059ffc3f12d11bd68938cfb1b4cd2462

MACHIN3 (MACHIN3) added a subscriber: Brecht Van Lommel (brecht).EditedMay 23 2020, 2:18 PM

I'm ready to close this but can somebody offer a suggestion what addons relying on non-supplied modules should do?
I can just append the path to sys.path when registering of course. @Brecht Van Lommel (brecht)?

Robert Guetzkow (rjg) added a subscriber: Robert Guetzkow (rjg).May 23 2020, 2:23 PM

In T71420 it was debated whether Blender would get it's own site-packages directory for the ones installed with the --user flag by setting PYTHONUSERBASE . Not sure if there's been a decision for these changes yet.

Installing the packages directly appears to be the way to go at the moment.

CansecoGPC (CansecoGPC) added a subscriber: CansecoGPC (CansecoGPC).May 23 2020, 5:32 PM

I was wondering why an add-on i was using didn't pick up it's dependencies, good to know.

Sadly the solution provided doesn't work, because as soon as i try to install with pip without --user flag on Linux this is what happens.

"Defaulting to user installation because normal site-packages is not writeable"

I hope the solution isn't launching Blender with root privileges.

MACHIN3 (MACHIN3) added a comment.EditedMay 23 2020, 5:38 PM

"Defaulting to user installation because normal site-packages is not writeable"

As it should be to be honest.

--user installation is definitely prefered IMO.

I just run this now, when registering my addon:

def verify_user_sitepackages():
    usersitepackagespath = site.getusersitepackages()

    if os.path.exists(usersitepackagespath) and usersitepackagespath not in sys.path:
        sys.path.append(usersitepackagespath)
Robert Guetzkow (rjg) added a comment.EditedMay 23 2020, 5:57 PM

@MACHIN3 (MACHIN3) @Brecht Van Lommel (brecht) Both certainly works, not sure what is currently recommended in the context of Blender. Having a proper and unified way of installing, managing and isolating dependencies would be nice. Perhaps that's a project I could get involved in once my thesis is done.

MACHIN3 (MACHIN3) added a comment.May 23 2020, 6:00 PM

It's a worthy cause for sure :) Good luck with your thesis and thanks for your input!

Robert Guetzkow (rjg) added a comment.May 23 2020, 6:07 PM

@MACHIN3 (MACHIN3) Thank you! No problem, you're welcome.

CansecoGPC (CansecoGPC) added a comment.May 23 2020, 6:13 PM
In T76993#937328, @MACHIN3 (MACHIN3) wrote:

"Defaulting to user installation because normal site-packages is not writeable"

As it should be to be honest.

--user installation is definitely prefered IMO.

I just run this now, when registering my addon:

def verify_user_sitepackages():
    usersitepackagespath = site.getusersitepackages()

    if os.path.exists(usersitepackagespath) and usersitepackagespath not in sys.path:
        sys.path.append(usersitepackagespath)

Just needed to import os, sys, and site and it worked. Thanks for the solution.

Robert Guetzkow (rjg) edited projects, added Python API; removed BF Blender.May 23 2020, 7:52 PM
Robert Guetzkow (rjg) added a project: BF Blender.
Campbell Barton (campbellbarton) closed this task as Archived.May 25 2020, 9:49 AM
Campbell Barton (campbellbarton) claimed this task.
Campbell Barton (campbellbarton) added a subscriber: Campbell Barton (campbellbarton).

This is working as intended, pass --python-use-system-env for previous behavior.

Robert Guetzkow (rjg) added a comment.May 25 2020, 10:10 AM

@Campbell Barton (campbellbarton) Since add-on developers can't expect that Blender is started with this command line flag, I assume that installing packages without the --user flag is currently the intended approach?

MACHIN3 (MACHIN3) added a comment.EditedMay 25 2020, 10:30 AM

This is working as intended, pass --python-use-system-env for previous behavior.

I can understand that, but honestly, this is a terrible thing to ask of addon users.

stephen leger (stephen_leger) added a subscriber: stephen leger (stephen_leger).EditedMay 25 2020, 2:05 PM

This is working as intended, pass --python-use-system-env for previous behavior.

Would rather say this is not working for average user anymore.

Create a site_package in user folder as working alternative out of the box and set the path according instead make a bit more sense than simply removing and let addon devs deal without.

%user%/scripts/site_package/

James (martiaus1028) added a subscriber: James (martiaus1028).Apr 14 2022, 9:05 PM

@MACHIN3 (MACHIN3) Thank you for the tip! did you include that snippet in the register function just once? and then you still pip install and just have it go to the user installation?

MACHIN3 (MACHIN3) added a comment.EditedApr 14 2022, 9:38 PM

Yes, I only run it once in __init__.py in register()

To install pip, I do

cmd = [pythonbinpath, ensurepippath, "--upgrade", "--user"]
pip = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

to install a module - PIL/Pillow in my case, I do:

cmd = [pythonbinpath, "-m", "pip", "install", "--upgrade", "--user", "Pillow==%s" % (version)]
pil = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

In both cases --user ensures the user site packages path is used.