I decided to try something else and I ended up with a workaround.
I added another collision to the player that interacts only with physics objects.
It’s at feet level and if it will overlap with object it disables it’s physics so it cannot be moved.
Of course it’s still possible to surf on objects if you try hard enough but shouldn’t happen by accidents. At least not that much.
I’m stil open to any better solutions because mine is not perfect and I think there is probably way easier one.