Yup… low and behold, the solution is always that simple.
*Edit: It was only 1-2 meshes that did not have proper collision set up. This is why the problem was hard to spot, as all others meshes were fine.
For people that might run into the same problem in the future:
Check the StaticMesh of your Static Mesh Components and check, at the top, for “Collision”.
Try and add a simple box collision if there isn’t one.
That fixed it. Thank you @Sklor