Texture Displacement Sculpting

Some things originally in this task have moved to T96223: PBVH image texture painting technical design and T96225: PBVH image texture painting implementation.

The idea here is to build on the above tasks, and make sculpt tools work with displacement textures and normal maps.

For some workflows this is a huge improvement. Game artist that rely on high poly sculpts and baking tools to create details in assets for game engines won’t need any of that anymore. They will be able to use a fully feature sculpting system to add the detail to the textures of the model directly.

This is obviously going to fail and create a lot of artifacts if the sculpted deformation is too big, but that is not the initial idea (that is also going to fail if the the high res sculpt and the low res mesh shape differ too much). The idea is to avoid using high poly sculpts to add surface detail when it is not necessary, and for surface detail, this approach should work fine.

I can’t evaluate the performance of this sculpting mode, but if we add UDIM support to the PBVH_TEXTURE (should not be hard to do, specially if we don’t support faces across different UDIM tiles) and the performance is OK, it could potentially allow to sculpt surface detail on models with a much higher resolution than any other solution based on mesh geometry and vertex deformation.

To implement this, the idea is to add a second buffer that stores the vectors to get the “deformed” 3D coordinates, and its normals. When iterating over the vertices in PBVH_ITER_BEGIN, the vd.co will be calculated by using the original 3D position of the vertex + displacement vector stored in the buffer. It should be possible to write this information to the texture to get a displacement texture and it should also be possible to write the normals calculated when the PBVH_UpdateNormals flag is processed to get a normal map.

Possible Issues

• I’m still not sure how this is going to work across seams. I think that if we add some extra padding pixels to the node which contains that face as if we were extending the face in 3D space it should be fine, but hard to know without trying it.
• I’m also not sure what kind of performance we can expect out of this, but even if performance is not the best, we can still try to code it to support some of its features, like sculpting information to the textures.

Technical Requirements

• Implement the neighbor iterator in the sculpt API
• Implement the deformed coordinates buffer and the normal map/vector displacement calculations
• Update the Sculpt API to read data from the coordinates buffer
• Add the Sculpt Mode UI to support textures
• Add normal map/displacement support to workbench (If possible, we can use EEVEE instead if that is hard to do)