If you set velocity directly, you may get problems if you actually collide with something.
As long as you’re free of collisions, you should be fine. This will give a very “arcade” kind of physics, which might be exactly what you want.
You can add listeners for collision/hit events, and update your own velocity calculation to bounce away from the collided plane. Easiest is to remove any component of velocity that moves into the inverse of the surface normal you collide with, BUT! There’s some gnarls with that if you wedge yourself into a narrow “V” crevice.
Another option is to save “recent known good non-collidiong position” and simply teleport to that position if you find that you’re colliding (and change velocity.)
Game physics really have two modes:
- Attempt to be as realistic as possible, accepting all the complexities of “real” physical dynamics combined with the imprecisions of collision detection and imperfect integrators.
- Attempt to set up all the velocities and positions the way you like it, and instead acceping all the impossibilities that can be caused by players, geometry, and your game rules disagreeing on reality.
Either way, it’s hacks all the way down 
The hacks just look different depending on your goal.