System Information
Operating system: Windows-10-10.0.17763-SP0 64 Bits
Graphics card: GeForce RTX 2070 SUPER/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 452.06

Blender Version
Broken: version: 2.91.0 Alpha, branch: master, commit date: 2020-09-29 00:11, hash: rBb150071ece73
Worked: Unknown, broken in 2.83.4 at the very least.

Short description of error
Some modes of the Vector Math node do not mute properly in Cycles. Specifically, the Dot Product, Distance, and Length modes are the ones affected. These modes all output greyscale values with the grey handle rather than vectors with the blue one. When muted, The red pass-through line indicates that the node should output the first input vector untouched, but instead it outputs a greyscale value. It appears to be the same greyscale pattern regardless of which mode was used. It does appear to be working properly in EEVEE. The problem is present in CPU, CUDA, and Optix modes, both in the viewport and in actual renders. It is present in 2.83.4 which is the oldest version I currently have access to.

Exact steps for others to reproduce the error
Use the Vector Math node on the Dot Product, Distance, or Length mode. Compare it unmuted, muted, and disconnected.
I've attaching a simple demo file, along with a image showing everything needed to reproduce the problem.

I only used the Dot Product mode in the examples, but it also works with the Distance and Length modes.