I actually ended up fixing the problem and completely forgot about this post.
I ended up reseting the linear damping and the angular damping of my trees.
Apparently this all started bc I migrated my project from UE4 to UE5 and it probably broke some values. I recommend reseting the linear damping and the angular damping, since it has been a while since I fixed this, I kinda forgot what I did besides that but I remember getting mad when it kinda fixed itself when I reseted those 2 values. Also I have LOCK POSITION X and Y enabled on the trees so they dont move that much when the physics are enabled. Now they vibrate a bit on the ground and thats it, barely noticable. Also check if the physical material that is applyed on the floor or wherever surface you are having problems and remove completly the physical material. I also had one when I was facing this problem, didnt add one again bc it was not needed for my game.