I too am using a wide angle lens 170° on a GoPro type action cam, I have been using video and mixing that out as jpg’s at 2-5 frames per second depending on length of the sequence. The results I have got are very good, the meshes are clean and detailed, I have not had any issues with stretched textures like you Brennmat, perhaps you might need to crop your source images to remove some of the distortion.
I have had a few issues with my setup, I do tend to get warped meshes on objects that are further away, for example the tops of buildings.
What would be the correct workflow when working with wide angle lenses and what distortion settings should we use?