It’s unclear what features you require. What you’ve listed it too general to tell. You might as well do it in UE4. You’d need to skim 5.4 / 5.5 release notes to see if anything pops up as a must-have / critical bug fix that bothered you. However, since you’re just starting, I’d stick to v5.5 since by the time you’re ready with Early Access release, we’ll be on v5.10 anyway.
Updating the project from 5.5 → 5.6 will be less painful than 5.3 → 5.6. Again, this depends how far you would have gone with the development.
It’s not that 5.3 has fewer / more bugs than 5.5. It merely has a different set of bugs 
The Physics Control component that I use a lot is under development, still experimental. And it worked better in 5.3 than 5.4/5.5 for my use cases, as an example - that’s be a reason to not switch / update.
I could not care about performance implication as this goes up / down between version releases. Make the game first, optimise later. There is no other way.
Now, if you had an entire studio and worked in a team, and had to decide now because you’re ready for rapid development and know you will not change versions, that’d be a different story altogether.