Does it matter what scale a game is created at?

you know, i thought about this for my company project.
I thought “what if i scale everything down 0.001%, i can have infinity world/landscape even at 1x”.
It didnt work out as planned lol. I had so many errors and bugs and physics was a nightmare.