Character and World scale

I’m wanting to create a “regular sized” town with a tiny character (like a toy soldier) . Would I shrink my character or scale my town up? Taking into consideration physics and such, I feel like it would be a scaled up town, but wanted to be sure