@kaymanv I have been dealing with the same issue for some time. This thread covers this issue as well and includes lots of options others have tried: UE5 gpu crashed or d3d device removed - #58 by Joshuab_artist
Switching to DX11 or Vulcan seemed to work, but like you, I lost out on too much functionality to use that solution.
The only other thing that I saw any success with was disabling “Support Hardware Ray Tracing” in the project. This resolved the device removed error for me. Unfortunately, my renders look way worse now (I’m doing cinematics). Perhaps that’s not essential for you or others, so you may wish to try that.