The reason your issue could be occurring is that you don’t have a variable setup for a drag offset with your Drag Drop Operation. With the drag offset applied I am not experiencing the issue you appear to be having. You can find a tutorial on making the drag-drop system which uses the drag offset here:
Creating Drag and Drop UI in Unreal Engine | Unreal Engine 5.1 Documentation