Maniphest T55954

Python/BMesh: Add/removing textures between renders causes memory leak/ram crash
Confirmed, NormalKNOWN ISSUE

Assigned To
None
Authored By
Charles West (charlesrwest)
Jul 12 2018, 9:06 PM
Tags
  • BF Blender
  • Python API
Subscribers
Bastien Montagne (mont29)
Charles West (charlesrwest)
Greg Izatt (gizatt)
Richard Antalik (ISS)
Zak Tab (zakleap)

Description

System Information
Operating system and graphics card
Ubuntu 18.04, Nvidia GTX 960M

Blender Version
Broken: blender-2.80-17bc0567268-linux-glibc219-x86_64

Working Blender Version
Blender 2.79b stable (ubuntu repository) with cycles

Short description of error
If you run the attached code and target it a directory with a large number of images, it will repeatedly:

  1. Load an image from file
  2. Set the image to be the content of a "Image Texture" node
  3. Delete/remove the image that use to be the image texture node
  4. Render

As it processes images, blender takes more and more memory. Eventually it will crash. If you leave out step 4, then it will not.

Exact steps for others to reproduce the error
Based on a (as simple as possible) attached .blend file with minimum amount of steps

Import python code, run

LoadAndRemoveTextures("/folderWithManyImages", 2000)

in the console window.


Code:

import datetime
import sys

import bpy
import bmesh
from mathutils import Vector
from mathutils import Euler
from bpy_extras.object_utils import world_to_camera_view
import random
import math
import json

import time
import os
import gc

def RemoveImageFromMemory (passedName):
    # Extra test because this can crash Blender.
    img = bpy.data.images[passedName]
    try:
        img.user_clear()
        can_continue = True
    except:
        can_continue = False
    
    if can_continue == True:
        try:
            bpy.data.images.remove(img, True)
            result = True
        except:
            result = False
    else:
        result = False
    return result

def CollectBackgroundImagePaths(sourceDirectoryPath, imageExtension):
    file_paths = []

    for root, dirs, files in os.walk(sourceDirectoryPath):
        for file_name in files:
            if file_name.endswith(imageExtension):
                file_paths.append(root + "/" + file_name)
    return file_paths

def SetBackgroundImage(backgroundImagePath):
    #Store reference to old image so it can be removed
    old_image_name = bpy.data.materials[0].node_tree.nodes.get("Image Texture").image.name

    #Load new image to be used
    new_image = bpy.data.images.load(filepath = backgroundImagePath)

    #Set the new image to be used
    bpy.data.materials[0].node_tree.nodes.get("Image Texture").image = new_image

    #Delete the old one from the file
    remove_succeeded = RemoveImageFromMemory(old_image_name)
    gc.collect()

    if not remove_succeeded:
        print("Error removing " + old_image_name)
    else:
        print("Removing " + old_image_name + " worked")

def LoadAndRemoveTextures(texturesFolderPath, numberToLoad):
    background_image_paths = CollectBackgroundImagePaths(texturesFolderPath, ".png")

    for example_index in range(0, numberToLoad):
        SetBackgroundImage(random.choice(background_image_paths))

        #If you alternate adding/removing images and rendering, it will eat all the ram on your system and crash
        bpy.ops.render.render( write_still=True )

Event Timeline

Charles West (charlesrwest) created this task.Jul 12 2018, 9:06 PM
Charles West (charlesrwest) added a comment.Jul 13 2018, 4:29 PM

Further investigation reveals that switching the renderer from Eevee to cycles also makes it stop eating all the ram, so it seems to be Eevee specific.

Bastien Montagne (mont29) lowered the priority of this task from 90 to 50.Jul 25 2018, 4:00 PM
Bastien Montagne (mont29) added a subscriber: Bastien Montagne (mont29).

Hrmmpfs… this is crashing here, in the scene_update call from render operator… depsgraph tries to access freed image ID, which is weird since relations are tagged to be updated in ID relink code.

No time to investigate further, but there are definitively issues here!

Charles West (charlesrwest) added a comment.Jul 25 2018, 4:10 PM

If I may ask, is there anything I can do to bump up priority on this? I'm doing some deep learning research that requires 3 million+ examples to be generated and switching from Eevee to Cycles with low samples has made rendering time go from 2.5 days to something like 10 days while reducing quality.

Can I post a bounty or something?

Zak Tab (zakleap) added a subscriber: Zak Tab (zakleap).Apr 9 2019, 7:55 PM
Zak Tab (zakleap) removed a subscriber: Zak Tab (zakleap).
Zak Tab (zakleap) added a subscriber: Zak Tab (zakleap).
Zak Tab (zakleap) added a comment.Apr 9 2019, 7:59 PM

I have found that this causes the Syncing memory to keep going up unbounded. On first render, memory peak is around 4000Mb, on the second it goes up to 8000Mb. This has completely killed my ability to render my scene.

Philipp Oeser (lichtwerk) added a project: Add-ons (BF-Blender).Nov 5 2019, 12:29 PM
Philipp Oeser (lichtwerk) renamed this task from Add/removing textures between renders causes memory leak/ram crash to Python/BMesh: Add/removing textures between renders causes memory leak/ram crash.Nov 5 2019, 12:32 PM
Philipp Oeser (lichtwerk) mentioned this in T71321: Python/BMesh: CDLayer access does not free memory.
Greg Izatt (gizatt) added a subscriber: Greg Izatt (gizatt).Dec 18 2019, 9:08 PM
Greg Izatt (gizatt) removed a subscriber: Greg Izatt (gizatt).
Greg Izatt (gizatt) added a subscriber: Greg Izatt (gizatt).
Dalai Felinto (dfelinto) added a project: Tracker Curfew.Dec 23 2019, 4:36 PM
Richard Antalik (ISS) changed the subtype of this task from "Report" to "Known Issue".Feb 10 2020, 10:26 PM
Richard Antalik (ISS) removed a project: Tracker Curfew.
Richard Antalik (ISS) added a subscriber: Richard Antalik (ISS).

Same "leak" happens with cycles as soon as I unpack list of loaded images in image editor (slowly and progressively - loading thumbnails?)
When I reload blend file images are gone, so they didn't have users.

Campbell Barton (campbellbarton) edited projects, added Python API; removed Add-ons (BF-Blender).Jun 19 2020, 10:42 AM
Campbell Barton (campbellbarton) moved this task from Backlog to Known Issues on the Python API board.Jun 19 2020, 11:06 AM