Selectively Cull objects by Camera

Hi, I have a setup, where VR and AR act together in the same space.
For performance purposes I’d like to cull (hide) some more complex objects.

This is trivial in Unity, but I cannot find a way how can I hide objects either by Tag or even by Distance.

GOAL Show only objects with a specific tag (yet still use their collisions …)

I found the “CullDistanceVolume” that did what I wanted for my AR setup, but I cannot selectively turn it off.
The idea was to turn it on for AR users and off for VR users, but even destroying the cull volume does nothing.

Is there any approach I can try?

[EDIT] I realised that “CullDistanceVolume” is not what I need as it only works for Static Meshes that are marked “static”.
Any help is greatoy appreciated. Thanks!