You could for example create Trigger Points inside and outside of the Cave to contorl the rain.
For Example. If you are 10 m in front of the Cave and the Rain Area is like 10m in all directions or whatever you can make it that as soon as the player enters that Trigger the Rain Stops following the Player.
And if you are going out somewhere else for instance you can as soon as the Player hits another Trigger change the Position of the Rain to the other loction so it rains when you are aproching the cave exit.
When you are out side you move through the Trigger Area which says it should stop follow you and into another one which again states that it should follow the player again and so on.
Quite complicated way with a lot of Triggers, but I don’t see any other way for you atm.