Texture painting brush uses two textures:

• Texture that overrides the color of the brush (brush.mtex)
• Texture Mask that acts as a masking reducing the effect of the brush. (brush.mask_mtex)

Sculpt/Vertex painting only supports a single mask.

• Texture that acts as a masking to reduce the effect of the brush. (brush.mtex)

This is confusing when we combine the texture painting and vertex painting in the same mode.

Cleanups to be done:

• BKE_brush_sample_tex_3d currently only uses mtex, perhaps a reason why sculpt is using mtex as mask_tex. masktex only has a 2d variant.

Approach

After doing some prototypes and detecting the different viewpoints about mtex by different developers, I was researching a solution that could sanitize this issue. Sanitation is required due to:

• Sculpt mode uses mtex for masking. Newly created brushes that could benefit from color masks are hard to implement in a maintainable fashion.
• Sculpt mode uses a concept of primary and secondairy mask texture.

Main idea is that all functions that access mtex/mask_mtex would access it via a wrapper.

struct MTexAccessor {
  MTex *color;
  MTex *mask;
};

Different tools/modes would then be responsible to create and fill this struct.
Specific implementations would only access the mask via the wrapper that is suitable for the functionality.
For example SCULPT_brush_strength_factor should pass MTextAccessor::mask to the BKE_brush_sample_tex_3d.

After this is done we could migrate per mode/tool to the actual brush mtex attribute that is designed for it.