Goal

To make the Paint Mode implementation feature complete we need to include an a way to mask the surface based on Cavity.

A first implementation of this has already been started in the sculpt-dev branch and can be further worked on.
In that case this feature will become a new automasking option in sculpt mode and and paint mode as well.

Proposal

The way we want to handle this is via the auto-masking options in sculpt mode. Since paint mode is based on the same code, the feature can be used for both modes.

Advantages:

• Can be toggled on the fly
• The auto-mask can be combined with other auto-masking methods without changing the painted mask.
• Masked areas can be changed non-destructively
• The auto-mask will update to the latest surface curvature

Settings

There are various controls that are needed to get the cavity mask users want.

• Falloff curve (Full control over which areas of the mesh curvature will be auto-masked)
• Inverted Cavity (An easy toggle to flip the effect of the auto-mask)
• Smoothing Steps
  • Currently called propagation steps. We could rename the setting to smoothing and keep calling it propagation in the tooltip and code
  • This setting must be exclusively tied to the cavity auto-masking, instead of using the regular propagation steps in the brush settings.

There are other design aspects to consider.

Performance

On high polycounts or when using a high amount of propagation steps it is likely to cause lag on each brush stroke.
@Joseph Eagar (joeedh) pointed out that D14272 and an API for temporary attributes would fix this.

Another boost could be to check if any important changes happened to the mesh in between strokes.
If vert positions haven't changed then the cavity auto-masking doesn't need to be recalculated on repeated strokes. So it would often only lag once.

The same could then be done for topology and face set auto-masking if the cursor is still targeting the same area.
Boundary auto-masking would also benefit from this.
This needs to be discussed and tested further.

Visualising the Cavity Mask

There needs to be some visual feedback on what is currently being auto-masked. Especially when we add fine control over propagation steps and what curvatures are masked via a falloff curve.

Since the automask is only generated under the brush cursor when doing a stroke, this becomes hard to implement.
If no solution is found, then a better way would be use an operator to convert the cavity automask into a regular mask attribute direectly.

Persistent Cavity Mask

There needs to be an option to keep the cavity mask persistent between strokes.

This would be easiest to implement with an operator to create a mask attribute based on the current cavity masking settings.
An operator like this could have multiple purposes:

• Use a persistent mask for detailed sculpting adjustments
• Visualise and edit more complex cavity masks
• Bake or export cavity masks for further workflows

We already have an operator like this called "Dirt Mask". This operator could be updated to use the same settings as the cavity automask.
For clarity it could then be renamed to "Mask by Cavity", with a tooltip that states that the Cavity automask settings are used for this operator.
Additional adjust last operation options should be made available too.