Git LFS and deleted assets reappearing

Happy new year!

I’ve been trying to get my head around this issue for a while, but can’t figure out a solution for it.
My project is set up using git with LFS configured.

Once I add an Actor to my Map and commit this change to git, I cannot permanently remove it afterwards. In Detail: When I close the editor and open it again later (not immediately after closing, seems this takes a certain amount of time) the deleted Actors reappear.

This is extremely annoying (e.g. in case of re-designing rooms and having nearly invisible - like glass- objects that act blockingly) and caused my quite some headache recently.

I know there’s been some people asking this before (see here and here. And here too), but never with a solution.

Does anyone have an idea how to get rid of this vexing behaviour?

Thanks in advance & cheers
Seb.