Ok so just to be up to date:

• The major bottleneck on most system is the overdraw.
• Geometry shader is a bottleneck but not that much. So not caching anything for now.
• We reduced the overdraw by extruding only relevant geometry.
• We made the algorithm more robust by adding depth fail method.
• The only remaining problem is visible if you have shadow casting non-manifold geometry and the camera is in shadow.

There maybe a solution to the last problem but I need to implement it first.

Ok so the last problem has been adressed.
So hopefully we have working shadows in all cases and it's not as heavy as it was.
If there is remaining problem it's a bug in the implementation and need to be reported.

Here is a list of some (all?) my commits on this subject.
rBaefc793a818aa8fa35e968b2be38e1b73909907d
rB60ddea7758128283f3ed3a15a2f8448691a811a8
rB975eac0b0794e6455a3fb642d4e1e0782cdd292e
rB0c9974c8cd96c1c089cb3f00fb782df7b8294d6c
rBe47e60a9b9692ed0e0fc0397e2841f9d547847c4
rB101c277e3d8241ad55f6343690b5d96a5e37d1f0