Hello
I have been using Reality Capture for a couple of days and I would like to give some feedback about my usage and learning how to use this software.
My background :
I am an amateur learning to do photogrammetry for fun. I have no background in making game nor visual effects. Reality Capture is the only photogrammetry software I’ve used.
I tied a couple a small test projects : I am mostly interested in scanning architecture, both inside and outside, I took some pictures using my mobile phone in a variety of conditions : outside under cloudy skies (ideal), outside a dusk (nice light but really dim), inside with a barely acceptable to moderate amount of light (no studio lights used).
I big difficulty I am encountering often is I don’t take enough pictures for the alignment to automatically work. I then have to spend an painful amount of time stitching manually the cameras with a huge amount of control points, to the point of sometimes failing and making my alignments break apart.
I tried using the RealityScan app. I love the augmented-reality feature that allows you to see where you took all the other pictures, but I found it to be limited due to the app’s insistence to connect to external servers, and the app hiding the pictures in a private folder making it painful to transfer files to my computer. The Realityscan is also limited in it’s ability to leverage my phone camera’s features. (Can’t use the wide angle lens, can’t change the compression format, etc…, these features require opening the phone’s camera app manually)
I found an alternative by using RealityCapture’s real time assistance feature on my laptop while taking pictures with the mobile phone.
Harware :
Camera (phone) : Samsung A54
Laptop : Asus Zenbook 14" with AMD IGP
Desktop : self assembled gaming pc, Ryzen 7800X3D, 32B ram, Nvidia RTX 4070 super
The laptop not CUDA capable, It’s a small and light device with an AMD IGP. So I only use it to run draft alignment when in the field and know whether I’m taking good coverage or missing key areas.
Then when I get back home I use the big desktop and do the rest of the work.
It took a couple of tries to get the mobile phone to connect to RealityCapture (windows firewall issue) but then I managed to make it work.
At first I tried taking pictures one by one, directly from the web app. Unfortunately I very quickly realized how slow and painful that was due to the excessive number of clicks required to take each and every picture. (open the camera mode, the phone’s camera returns to default configuration, so I have to reset the zoom, focus, and other settings every time, actually take the picture, confirm that yes you want to keep it, repeat).
I found a much nicer alternative by using the phone’s camera directly and then adding files in bulk using the file selector.
This way while the laptop process the draft alignment, I’m taking the next series of pictures.
Usually, the alignment is finished while I’m still photographing, so the laptop’s CPU idles a little between the bursts of processing. Allowing it to cool down and avoids burning through the battery too fast.
The process works quite well as long as the pictures align.
When the pictures don’t align, then it’s kind of hit and miss.
The we app does not provide controls to switch between components. It only displays the component with the most cameras. If I manually switch to other components on the computer, the app automatically switches back within seconds.
As a result, as long as I rapidly fix the alignment by taking a couple of missing-link pictures, I can get back on track. But If I repeatedly fail at finding the locations of the missing pictures, then I’m in the fog. I take more and more pictures in the blind without guidance often taking many overlapping pictures in the same place and orientation which doesn’t help.
It would be nice if the component auto-switch could be temporarily overridden, just to review the over components without interruption.
At the moment, the solution I had to come up with was to turn off the real time assistance while thinking and turn it back on (and re-connect on the phone) once I’m ready to make a new attempt. It’s tedious.
It would be nice to have a bridge between RealityCapture and Realityscan : you’d be able to physically see where the existing pictures are in augmented reality and more easily identify the missing links.
Despite, that, when I figured out my missing shots, I’ve had fun scanning using this workflow.
An issue I encountered was the progressive slowing down of the alignment as the picture count slowly but surely increases.
I figured out that I could reduce computation by manually selecting the early pictures in RealityCapture and change their “Feature source” to use “merge using overlap” only and lock down their positions. And only leave full computations to the latest pictures.
But eventually, as the picture count increases I always reach a point when the next alignment starts to take increasingly longer and longer, 30s, a minute, and more (It’s just a laptop, not a server farm) and then think about whether I’m done or continue by saving and starting a new session for the rest of the scan. And then hope they’ll align when I attempt to align then once back home on the big computer.
I wish there was an easier way to to multi-part scans for large interiors by easily switching between different sections / rooms / buildings, while still making sure the parts assemble during shooting.
Other suggestions :
Point cloud orientation : in my first tests, RealityCapture displayed the point clouds with a completely random orientation. Tilted, upside-down, sideways, it’s messy. You have to turn the image around every time because the view resets on each new version of the alignment. Things can get really hairy when the point cloud gets displayed sideways due to the way the rotation and translation controls are setup (gimbals lock ! can’t look up or down). I found that when disabling GPS location, this get significantly better. After only a couple of pictures, RealityCature figures out the correct orientation for both RealityCapture and the web app displayed.
I have seen messages from other users who have the same issue of having their point could orientations all over the place. I suggest changing RealityCapture’s default setting to ignore geolocation.
BUGs found :
Bug 1 : my phone camera app has HEIF compression, the phone does not provide any “quality” setting, It’s just “jpeg or heif”.
I didn’t spend hours comparing picture quality in detail, at a quick glance the two options look similar to me (looked on the phone’s screen with a little bit of zoom), but the file sizes are drastically smaller. So I would like to use this feature.
When taking pictures within the web app, the phone automatically disables heic and uses jpeg. But when uploading from the file browser, the real time assistant web app accepts the heif files, and sends then over to the computer, the files do land in the session’s _data folder, but RealityCapture does not add them to the project automatically. It only does so when using jpg compression.
Bug 2 : when sending files in bulk from the web app to the computer, RealityCapture does not index them in the order the picture were taken. As a result browsing through pictures using the arrow keys (left/right keyboard shortcuts) does not follow the correct order.
I think this is more a problem with the overall behavior of the previous/next shortcut, not just an indexing thing. I have trouble using this shortcut all over the app because it doesn’t do when I think it intuitively should be doing.
Bug 3 : camera FOV
I haven’t found if there is any FOV control in the web app, but it looks like there is some kind of automatic FOV adjustment. I’ve had a couple of times when the FOV went super-ultra-wide when loading an updated component. After a couple of pictures and a new version, the FOV fixed itself on it’s own. Couldn’t figure out what provoked this weird bug, nor what fixed it.
Bug 4 : after finished uploading files in bulk, the status indicator in the web app often does not display that computer is processing the alignment. It just says “ready”, like nothing is happening.
If I reload the page, then the status indicator actually reports that the computer is working on alignment (and provides the progress percentage).