System Information
Operating system: Windows-10-10.0.18362 64 Bits
Graphics card: GeForce GTX 1070/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 432.00

Blender Version
Broken: version: 2.81 (sub 16), branch: master, commit date: 2019-11-20 14:27, hash: rB26bd5ebd42e3

Short description of error
When disabling a boolean modifier in the viewport for performance reasons, there is still a considerable slowdown in animation playback unless the modifier is deleted (much smaller slowdown than with it enabled, but still a considerable slowdown).

Exact steps for others to reproduce the error

1. Create two meshes to use for the boolean modifier (at least one should be dense enough to cause a slowdown when boolean is used)
2. Add the boolean modifier to one of the meshes
3. Disable modifier in viewport and notice how there is still a slowdown
4. Remove the modifier completely and notice how there is no more slowdown.

Here is an example file