You are 100% right. It depends on your needs and you will always lose details if you decimate, even if it is only a tiny amount.
Just tested this with a 1x1 m stone for a floor, i endet up with 2 million polys.
Does it look good? Yes but will you see this tiny details in a game? i guess not.
And how does this add up if you use this for every pice in a level. How long will be loading times?
I mean this single little stone is 15mb compressed size.
But i think this is more for a own topic for a discussion about.
In the end it is really great to have nanite. You can just do insane amount of details.