What I actually did was unchecking ‘simulate physics’ option. This way i have to detect and handle all collisions with non-physic objects myself. Collisions with physic objects are detected(OnHit event) and I just have to reflect the velocity. 
There are some minor issues with that approach(sometimes there are weird reactions to collision), but it’s acceptable.
+I cannot set friction to 0 on all surfaces on the level, because apart from the ball I also have physic objects which I want to behave correctly.