Not my area of expertise but I’ll try to chip in.
@miguline What if you run Pawn sensing in the camera owning pawn? It could be as ham-fisted as this pseudoscript:
NPCs do not execute expensive script unless they’ve been sensed.
There may be some nuances / considerations regarding the scene setup. You say a small village - what’s our PoV, though? You’d want to avoid a springarm as it’d offset PoV for an actor-attached component. Ideally, you’d have a camera Pawn and use that as a camera.