Maniphest T94615

Blender 3.0 consistently crashes when RSS nears 5GiB on Linux (easy to trigger with video playback)
Closed, Resolved

Assigned To
Jesse Yurkovich (deadpin)
Authored By
George Hawkins (ghawkins)
Jan 3 2022, 8:15 PM
Tags
  • BF Blender
Subscribers
George Hawkins (ghawkins)
Jesse Yurkovich (deadpin)

Description

System Information
Operating system: Ubuntu 20.04.3 LTS
Graphics card: GeForce RTX 2060

Blender Version
Broken: 3.0.0, f1cca3055776, master, 2021-12-02
Worked: 2.93.7, a5b7f9dc9011, master, 2021-12-14

Note: I also downloaded the December 29th 3.0.0 daily build from https://builder.blender.org/download/daily/ but it has the same hash as the official release, the only difference seems to be the build date:

$ ./blender --version
Blender 3.0.0
	build date: 2021-12-29
	build time: 00:34:14
	build commit date: 2021-12-02
	build commit time: 18:35
	build hash: f1cca3055776
	build platform: Linux
	build type: release

Short description of error

Any action that pushes Blender's RSS near the 4.7GiB mark consistently causes Blender to crash on my very generic Ubuntu 20.04.4 LTS system (with all apt updates applied and the latest 470.86 proprietary Nvidia driver version).

E.g. if I play a video file, I can watch the RSS gradually grow (in top or htop) as the frame count rises and then as it reaches around 4.7GiB it crashes.

No such issue is seen if I switch back to a 2.93 version of Blender and do exactly the same thing.

Exact steps for others to reproduce the error

For a given video and a freshly opened copy of Blender, the frame at which it crashes, on playback, is entirely consistent.

E.g. for the 4K Footage.mp4 file here (this is just footage that accompanied this Blender Daily video on YouTube), I can consistently play from frame 1 to frame 129 (using left-arrow to step frame-by-frame as I approach 129) and then Blender crashes as I step onto frame 130.

The frame I can reach varies depending on the file used, e.g. I can get much further in a typical 1080p file. But if I watch what's happening with htop then the failure always seems to occur as the RSS goes just a little above 4700MiB.

E.g. here you can see that Blender's RSS is 580MiB before I started playback:

By frame 129, RSS has grown to 4703MiB:

Stepping onto the next frame caused it to crash (and it consistently crashes on just that frame). As you can see from the screenshots, the system has plenty of free memory and is not heavily loaded.

If I run Blender with strace then the last few lines I see are:

read(28, "\301i2\325\251\263xu\215\267\206Y\357\207\324\301\4.\347\207M\333\332(RR\207\353\331NT\242"..., 32768) = 32768
futex(0x7f578aed7a04, FUTEX_WAKE_PRIVATE, 1) = 1
madvise(0x7f5766c0a000, 33558528, MADV_DONTNEED) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x10} ---
write(1, "Writing: /tmp/blender.crash.txt\n", 32Writing: /tmp/blender.crash.txt
) = 32

The full strace output is here:

The blender.crash.txt is here:

Exact steps:

  • Start Blender and select VFX.
  • In the middle Movie Clip editor (the one with mode Tracking and view Clip), click Open and select a downloaded copy of the Footage.mp4 file linked to above.
  • Press Space to start playback and then, as the frame count nears 129, pause and start to step forward one frame at a time with cursor left-arrow.
  • Everything should go fine up to including frame 129 but on stepping onto frame 130 Blender crashes.

I can reproduce this with any reasonably long video clip - the frame at which Blender crashes depends on the video (4K videos blow up much earlier than 1080p videos) but the frame at which it crashes is consistent for any given video, i.e. I can get it to crash repeatedly at exactly the same point.

Related Objects

Mentioned Here
T93479: 3.0 Potential candidates for corrective releases
rB7061d1e39fea: Fix T92740: Missing lock around the image CacheLimiter
rBea8d749587dd: Cleanup: Rename ObjectValue to DictionaryValue (Serialization).
rBf1cca3055776: Blender 3.0 - version bump -> release
rBa5b7f9dc9011: Version bump: 2.93.7-release

Event Timeline

George Hawkins (ghawkins) created this task.Jan 3 2022, 8:15 PM
George Hawkins (ghawkins) added a comment.EditedJan 3 2022, 8:25 PM

I just tried the latest 3.1 alpha for Linux - blender-3.1.0-alpha+master.ea8d749587dd-linux.x86_64-release - and I no longer see this issue. So, I guess it has probably already been covered by a previous bug report.

Hopefully, whatever the underlying fix is, it will soon appear in the next stable version, i.e. 3.0.1. As noted in my original bug report it is not fixed in the latest 3.0.0 daily build (however, this daily seems to have exactly the same hash as the Dec 2nd 3.0.0 stable release, i.e. it's got no additional commits over the official 3.0.0 release).

Update: I spoke too soon, see next comment.

George Hawkins (ghawkins) added a comment.Jan 3 2022, 8:41 PM

Interesting - neither 2.93.7 and 3.1.0-alpha+master.ea8d749587dd crash on playing back Footage.mp4 but they both hang on the frame where 3.0.0 crashes if I try to track - ctrl-T - to that frame:

Here's the .blend file (you'll also need the Footage.mp4 file linked to in the original bug report):

Jesse Yurkovich (deadpin) added a subscriber: Jesse Yurkovich (deadpin).Jan 3 2022, 11:02 PM

The MEM_CacheLimiter_unmanage crash as seen in your crash file was fixed by rB7061d1e39fea7495e787a071b83757e3dc0d61a7 and is already on the list of patches to include in 3.0.1 if/when that happens (T93479)

Beyond that, I'm unable to reproduce the hang here locally on 3.1. I was able to complete the track for all frames. If it's reliable for you it would be best to create a new report, just for the hang, so things are clear for other folks looking through the bugs.

George Hawkins (ghawkins) added a comment.Jan 4 2022, 9:16 AM

Please consider this bug closed from my perspective (I don't see any way for me to close it directly).

Thanks for the quick response and apologies for not being better at analyzing blender.crash.txt to see that it was already covered by another bug.

I agree that it makes sense to split out the tracking issue - will look into it more and see if there really is an issue and if I can clearly summarize it in a new bug.

Jesse Yurkovich (deadpin) closed this task as Resolved.Jan 4 2022, 11:06 AM
Jesse Yurkovich (deadpin) claimed this task.

No worries, it would have been a bit of a pain to find the already-fixed issue really. I'll resolve this task but if you can reliably reproduce the hang, please do file a new report.

George Hawkins (ghawkins) added a comment.Jan 5 2022, 8:29 PM

Thanks, Jesse. The "hang" was actually the result of hitting the video-sequencer cache's default limit of 4GiB. When I upped it to 16GiB the track ran through to the finish.

I'm not quite sure why tracking through a video file (or simply playing it) should cause Blender to fill up and then grind to a halt once you've read thru e.g. 129 frames of 4K video in my case.

Surely the point of a cache is that you start throwing away the older data (i.e. the earlier frames) once full and that, ideally, ejection from the cache shouldn't be really expensive?

But at the moment, once the cache is full that seems to be Blender pretty much done for - it will grind on from that point but extremely slowly (but with htop showing very low CPU usage - so it's not maxing out there).