Grab object issue

I am working on a 3d side scroller and I am able to have my character grab objects and pull them around the map, but I am unable to push them. The object kind of gets stuck on the character when I try to push it. If I let the object go and jump to the other side I can pull it in the direction I tried to push.

I am just using the basic grab blueprint on my character.

Any ideas?