Hi, I can't reproduce the crash you describe using your project. This is the result of noise reduction in your project after sampling 800 rounds, I did not change any settings in your project. Maybe you can try re-rendering a few times?

Here are the files.

In T101529#1426416, @liang yunfei (LemonBranny) wrote:

Hi, I can't reproduce the crash you describe using your project. This is the result of noise reduction in your project after sampling 800 rounds, I did not change any settings in your project. Maybe you can try re-rendering a few times?

renderresult.exr514 KBDownload

It doesn't crash all the time. You can try to duplicate the sphere in the scene 2 or 3 times and render again.

I have confirmed this problem. you can create a new file, make a Suzanne as glass, make a ground, and then let the path-guiding method render, depending on your computer hardware, the software will crash after a fixed number of sampling. The number of samples to crash varies with the hardware of the computer, good computers may crash after a very high number of samples, bad computers will crash after fewer samples, but they will eventually crash. Or you can use this project :

Liang, you can move the camera around and keep training data. If anything else changes, such as moving an object or changing a setting, you lose training data.

Thank you @liang yunfei (LemonBranny), for your analysis, this helps a lot.
This pretty much helped to figure out the cause of the problem, which seems to be a wayyy large bound for the spatial guiding structure.
At these high sample counts and with unlimited training, the structure would refine continuously, which causes the memory increase.
I will post a fix soon.
As a side note, unlimited training does not bring you many performance benefits (in terms of better sampling) and only leads to a performance decrease due to the additional training time.
Unfortunately, we did not find a good automatic way (which is robust in production) to determine the optimal number of training iterations you need.
As a compromise, we decided to give the artist the option to set the number of training iterations (usually 64-128 are enough).

@Andreas Resch (AndreasResch) could you share more information about your system like:

• overall memory size (RAM)
• used memory when loading the scene (no rendering)
• used memory when rendering (without guiding)

I am curious if your bug is related to what @liang yunfei (LemonBranny) reported since after 400spp the guiding structure should not be large.

RAM should not be an issue.

*) The system has 64GB of RAM
*) With path guiding the RAM jumps from 10,8GB to 11,7GB and then crashes.
*) Without the RAM it reaches a maximum of 11,3GB during rendering.

@Andreas Resch (AndreasResch) OK, then your problem is probably not related to the memory problem reported by @liang yunfei (LemonBranny)
At least on Linux I am not able to replicate your bug. I am getting a lot of warnings that some blend files are missing:

Read blend: /data/sherholz/Develop-Blender/scenes/PathGuidingTest_01 (2)/PathGuidingTest_01.blend
Info: Read library:  '/data/sherholz/Develop-Blender/scenes/Work/Verschiedenes ... P-T/Schmidt Spiele/Mille Fiori/_BLENDER/MilleFioriErw_Spielszene_01.blend', '//..\Work\Verschiedenes ... P-T\Schmidt Spiele\Mille Fiori\_BLENDER\MilleFioriErw_Spielszene_01.blend', parent '<direct>'
Info: Cannot find lib '/data/sherholz/Develop-Blender/scenes/Work/Verschiedenes ... P-T/Schmidt Spiele/Mille Fiori/_BLENDER/MilleFioriErw_Spielszene_01.blend'
Info: LIB: Image: 'Bildschirmfoto 2021-06-09 um 11.24.jpg' missing from '/data/sherholz/Develop-Blender/scenes/Work/Verschiedenes ... P-T/Schmidt Spiele/Mille Fiori/_BLENDER/MilleFioriErw_Spielszene_01.blend', parent '<direct>'
Info: Read library:  '/c/Programme/Vshade/Vshade - Color.blend', 'C:\Programme\Vshade\Vshade - Color.blend', parent '<direct>'
Info: Cannot find lib '/c/Programme/Vshade/Vshade - Color.blend'
Info: LIB: NodeTree: 'Vshade - Absorption Calc. (Glass)' missing from '/c/Programme/Vshade/Vshade - Color.blend', parent '<direct>'
Info: LIB: NodeTree: 'Vshade - Fresnel (Lite)' missing from '/c/Programme/Vshade/Vshade - Color.blend', parent '<direct>'
Info: Read library:  '/c/Program Files/Vshade/Vshade - Shader - Cycles.blend', 'C:\Program Files\Vshade\Vshade - Shader - Cycles.blend', parent '<direct>'
Info: Cannot find lib '/c/Program Files/Vshade/Vshade - Shader - Cycles.blend'
Info: LIB: NodeTree: 'Vshade - Solid Shader (Lite)' missing from '/c/Program Files/Vshade/Vshade - Shader - Cycles.blend', parent '<direct>'
Info: Read library:  '/c/Programme/Vshade/Vshade - Shader - Cycles.blend', 'C:\Programme\Vshade\Vshade - Shader - Cycles.blend', parent '<direct>'
Info: Cannot find lib '/c/Programme/Vshade/Vshade - Shader - Cycles.blend'
Info: LIB: NodeTree: 'Vshade - Glossy Shader (Basic)' missing from '/c/Programme/Vshade/Vshade - Shader - Cycles.blend', parent '<direct>'
Info: LIB: NodeTree: 'Vshade - Micro Roughness' missing from '/c/Programme/Vshade/Vshade - Shader - Cycles.blend', parent '<direct>'
Info: LIB: NodeTree: 'Vshade - Refraction Shader' missing from '/c/Programme/Vshade/Vshade - Shader - Cycles.blend', parent '<direct>'
Info: LIB: NodeTree: 'Vshade - Refraction Shader (Base)' missing from '/c/Programme/Vshade/Vshade - Shader - Cycles.blend', parent '<direct>'
Info: LIB: NodeTree: 'Vshade - Set Shadow' missing from '/c/Programme/Vshade/Vshade - Shader - Cycles.blend', parent '<direct>'
Info: Read library:  '/data/sherholz/Develop-Blender/scenes/Work/Verschiedenes ... P-T/Schmidt Spiele/Mille Fiori/_BLENDER/MilleFioriErw_Spielszene_01.blend', '//..\Work\Verschiedenes ... P-T\Schmidt Spiele\Mille Fiori\_BLENDER\MilleFioriErw_Spielszene_01.blend', parent '<direct>'
Warning: Unable to open '/data/sherholz/Develop-Blender/scenes/Work/Verschiedenes ... P-T/Schmidt Spiele/Mille Fiori/_BLENDER/MilleFioriErw_Spielszene_01.blend': No such file or directory
Info: Cannot find lib '/data/sherholz/Develop-Blender/scenes/Work/Verschiedenes ... P-T/Schmidt Spiele/Mille Fiori/_BLENDER/MilleFioriErw_Spielszene_01.blend'
Info: LIB: Image: 'Bildschirmfoto 2021-06-09 um 11.24.jpg' missing from '/data/sherholz/Develop-Blender/scenes/Work/Verschiedenes ... P-T/Schmidt Spiele/Mille Fiori/_BLENDER/MilleFioriErw_Spielszene_01.blend', parent '<direct>'
Info: Read library:  '/c/Programme/Vshade/Vshade - Color.blend', 'C:\Programme\Vshade\Vshade - Color.blend', parent '<direct>'
Warning: Unable to open '/c/Programme/Vshade/Vshade - Color.blend': No such file or directory
Info: Cannot find lib '/c/Programme/Vshade/Vshade - Color.blend'
Info: LIB: NodeTree: 'Vshade - Absorption Calc. (Glass)' missing from '/c/Programme/Vshade/Vshade - Color.blend', parent '<direct>'
Info: LIB: NodeTree: 'Vshade - Fresnel (Lite)' missing from '/c/Programme/Vshade/Vshade - Color.blend', parent '<direct>'
Info: Read library:  '/c/Program Files/Vshade/Vshade - Shader - Cycles.blend', 'C:\Program Files\Vshade\Vshade - Shader - Cycles.blend', parent '<direct>'
Warning: Unable to open '/c/Program Files/Vshade/Vshade - Shader - Cycles.blend': No such file or directory
Info: Cannot find lib '/c/Program Files/Vshade/Vshade - Shader - Cycles.blend'
Info: LIB: NodeTree: 'Vshade - Solid Shader (Lite)' missing from '/c/Program Files/Vshade/Vshade - Shader - Cycles.blend', parent '<direct>'
Info: Read library:  '/c/Programme/Vshade/Vshade - Shader - Cycles.blend', 'C:\Programme\Vshade\Vshade - Shader - Cycles.blend', parent '<direct>'
Warning: Unable to open '/c/Programme/Vshade/Vshade - Shader - Cycles.blend': No such file or directory
Info: Cannot find lib '/c/Programme/Vshade/Vshade - Shader - Cycles.blend'
Info: LIB: NodeTree: 'Vshade - Glossy Shader (Basic)' missing from '/c/Programme/Vshade/Vshade - Shader - Cycles.blend', parent '<direct>'
Info: LIB: NodeTree: 'Vshade - Micro Roughness' missing from '/c/Programme/Vshade/Vshade - Shader - Cycles.blend', parent '<direct>'
Info: LIB: NodeTree: 'Vshade - Refraction Shader' missing from '/c/Programme/Vshade/Vshade - Shader - Cycles.blend', parent '<direct>'
Info: LIB: NodeTree: 'Vshade - Refraction Shader (Base)' missing from '/c/Programme/Vshade/Vshade - Shader - Cycles.blend', parent '<direct>'
Info: LIB: NodeTree: 'Vshade - Set Shadow' missing from '/c/Programme/Vshade/Vshade - Shader - Cycles.blend', parent '<direct>'
Warning: 4 libraries and 9 linked data-blocks are missing (including 0 ObjectData and 0 Proxies), please check the Info and Outliner editors for details

Could this be the reason why me and @liang yunfei (LemonBranny) can not replicate your observed crashing behaviour?

@liang yunfei (LemonBranny) I also want to answer your two questions:
1. Memory problem:
This was a bug and will be fixed soon

2. Training at low resolution vs high resolution:
The behavior you reported is expected.
Path guiding learns an approximation of the light distribution in the scene by using information
from the paths of previous rendering iterations. Therefore higher resolutions will provide more information (samples)
than lower resolutions, since more paths are traced (one per pixel) per rendering iteration.
As a result, the guiding structure has higher quality than if you would run it at a lower resolution.
Since the learned guiding structure is directly used after the first rendering iteration, the sampling in the second iteration is better, and
therefore the quality of the updated structure becomes even better.
this is also the reason why a smaller number of high-resolution training passes usually outperform many training passes at lower resolutions.

@Sebastian Herholz (sherholz) Thank you very much for your patient reply! After hearing you say that 64-128 training iterations are usually sufficient, I realized that resolution is also one of the important factors in training results. I tested training with high resolution in my project, and with just 128 iterations I got the same results as training with low resolution for thousands of iterations. However, some smaller caustic rays still cannot be calculated. I know that it is not designed to calculate caustics, but in some cases, the original cycle cannot calculate caustics at all, so this method can sometimes improve the calculation speed by tens of thousands of times. Your explanation is wonderful, please be sure to write it in the blender manual after this feature is officially added to blender.
However, I still have a question, path-guiding sometimes introduces a lot of very bright color noise, and these noises do not become significantly darker as the number of samples increases.I wonder if this is inevitable or if it can be optimized in a future version.
Path-guiding creates a lot of bright colored noise, which makes denoise impossible:

@liang yunfei (LemonBranny) how many samples did you use for the path guided version?
There are two things:

1. I saw that you played around with DEBUG parameters like guiding probabilities.

And you set them from 0.5 to 1.0. These parameters are for debugging only and should not be changed.
0.5 means that you do 50% BSDF sampling and 50 guiding, which is a stable conservative ratio so that the path tracer
trusts the guiding cache but still explores the scene to capture any contribution not observed by the cache.
If you go full 100% into guiding it can happen that you mis or hardly under explore important parts of your scene.

2. I assume you use less samples in the PG image.

Since guiding has to learn from over time, early samples will still have some fireflies or firefiles are generated
from sampling decisions which are not or can not be guided (e.g., reflect/refract on glass).
These fireflies still need a lot of samples to be average out like in the plain 1.6M rendering.

aA solution would be to either use more indirect radiance clamping (which would make the image darker) or
Cycles would need to add some more inteligent firefly removal approach.

@Sebastian Herholz (sherholz) Thank you very much!The image that uses path-guiding has only 16000 samples. It is already 2:30 in the morning here in China, so I have to go to bed. I will get up tomorrow morning and test the method you mentioned,and I will give you a reply as soon as possible.

In T101529#1429262, @Sebastian Herholz (sherholz) wrote:

@Andreas Resch (AndreasResch) OK, then your problem is probably not related to the memory problem reported by @liang yunfei (LemonBranny)
At least on Linux I am not able to replicate your bug. I am getting a lot of warnings that some blend files are missing:

But I'm not actively using any of those linked any more in the test scene. That's one of those cases where Blender isn't that great at deleting stuff that isn't used any more.

Here's a new file that still crashes. It's basically the same scene but without the linked items (that weren't used in the scene anyway).

@Sebastian Herholz (sherholz) For the first thing you said, I found that setting the guiding probabilities to 1, at least in this project of mine, resulted in a less noisy and sharper caustics image than setting the guiding probabilities to 0.5. In fact I found in most of my test projects that setting the guiding probabilities to 1 resulted in less noise and sharper caustics.
In addition, I forgot to mention that the reason why my test project has colored noise is because my glass objects all use my dispersion shader. If you use blender's default shader, the color of the noise should be solid.
Guiding probabilities 1, trainning 1024 , 2000 samples:

@Andreas Resch (AndreasResch) The project you've given now has allowed me to successfully reproduce the crash, and I'll be looking into the project settings that might be causing the crash.
Edit:
I have found the cause of the crash: in this project, using both glass and volume shaders caused the crash, using either the glass alone or the volume shader alone did not cause the crash.

@Andreas Resch (AndreasResch) Thanks for the update, I can now also reproduce the crash.
I will have a look at it after the weekend.

@Sebastian Herholz (sherholz) Please can I ask why path guiding doesn't work in such a project?
Cycle GPU 2000 samples:

@Andreas Resch (AndreasResch) I wrote some patches which should solve your problem.
Could you please check the current experimental build:
https://builder.blender.org/download/experimental/cycles_path_guiding/

@liang yunfei (LemonBranny), I think the scene is just too hard to render.
On a first test, I think two things could be the problem here:

1. Since path guiding learns from previous rendering iterations, there has to be

a high enough initial probability of finding these caustic paths so that path guiding can learn from them.
Here this initial probability might be too low.

2. No all decisions are guided, mainly the ones on diffuse surfaces. The "reflect or refract" decision is un-guided and therefore based on the Fresnel term.

It could be that in this scene, Fresnel things it is best to reflect (based on the material), but it would be better to refract (based on the indirect lighting).
If this is the main cause of the noise than path guiding can not help here.

@Sebastian Herholz (sherholz) Thank you very much, your answer is so detailed. I still have a question, that is, in your plan, path guiding will support refraction shader and specular shader in the future, will path guiding be able to help sample such projects at that time?

In T101529#1430066, @Sebastian Herholz (sherholz) wrote:

@Andreas Resch (AndreasResch) I wrote some patches which should solve your problem.
Could you please check the current experimental build:
https://builder.blender.org/download/experimental/cycles_path_guiding/

It works fine now. No more crashing.