By adding animation to the modifier stack you effectively make it so the modifier stack depends on time (since animation depends on time, and modifier stack depends on animation).

There is no tracking of state happening, so when input (such as time) changes the update is propagating through the entire hierarchy, without any checks done from the previous evaluation.

So it is expected that modifier stack is re-evaluating in this case.

Solving this is not impossible, but it would require changing animation system and the way how updates are propagating. And it's really tricky to make reliable without performance regression for the case when real animation is involved.

What you can try doing is to have a pre-update handler and disable visibility of modifier if it's in a state which you don't want it to (as in, don't just assign the value, compare the value first to avoid the block being tagged for update).

This affects playback performance at the studio!

If you have static object where you want to control something on per-shot basis then just create a rig for that object and use driver from that rig. And don't animate the rig.

If it's an object which is belongs to an animated character, then it does need to be re-evaluated and there is no way around it.

All in all, such issues are touching fundamental designs which can not be tackled in a nearest future and hence are handled outside of the bug tracker.