PhysX is certainly good enough to create a playable billiards game, but like all other real time physics engines it’s not perfect so just keep that in mind.
The engine itself does not care what units you work in as long as everything is scaled properly with your units (gravity and other foces for example). Small objects are known to pass through other objects if they travel too fast. To combat this you can enable ‘Continous Collision Detection’ (CCD) for your spheres.
Also try to avoid having planes as collision objects. Give them some depth, or an invisible box collision component.
I know in UE3 you could also change the steps the physics engine takes to solve collision for greater precision, I would assume there’s a setting for this somewhere in UE4 as well (World settings probably :)).
One last thing to remember is that physics games have a sweetspot where they behave like you want them to, and anything outside of that causes catastrophic behavior. So even though everything is set up right except a few tweaks to mass and forces it can look like it’s completely broken.