Thanks for the hint with the pdb file location. The fix for me was to run the debugger as you did, and then in VS go to Debug > Windows > Modules, and look for my project dll (UE4Editor-YourProject-Wing64-DebugGame.dll), right-click on it select load symbols, and manually select the pdb file.
I think the source of the issue has something to do with my project being in a mounted folder (I mapped an older drive to C:\Slowdrive and put my UE project there). When I was in the debug modules window I saw that some where located in C:\SlowDrive… and some were in \Device\HarddiskVolume5…, so I think the build is getting confused between the two.
I followed the instructions here