Sounds like a programming problem.
You’re building up a lot of X ( something ), and it’s slowing down the system.
It’s probably something you’re creating, spawning etc. Take a look at the code, maybe.
It’s also worth keeping an eye on the outliner. Does the number of actors just keep going up?