So I thought I would do an update to what happened. After trying many ways of converting the old project, migrating assets, removing foliage and starting again, nothing seemed to work. In the end the best thing to do is start a new project in UE5 or keep working in 4.
If you really want to keep an old project but use it in UE5, just delete all the foliage and reimport all the assets and re paint them in UE5.
Also if you find the migrating UE4 project has bad FPS in 5, its mainly because of Lumen, you can use the old ray tracing system.