After trouble shooting and then going through to reproduce the problem to post what I was doing, I found the issue. Kind of embarrassing error on my part. The models textures had a image hooked up to the world position offset. On the larger models the change was so small that you couldnt even notice it. But when your dealing with models that get very small the offset appeared much much greater due to them being small.
So there is no issue afterall. Just a error on my part.