Workflow Process - Interior Image Alignment

Development Asset Creation
,
1

Hi Everyone,

I thought I would put together some experience I had with trying to align photos of interior buildings. Most of the information is available in other posts but it seems to me like it needs to be pieced together a bit.

Keep in mind the projects I am working on are to bring real house interiors into VR. Valve has an amazing guide here but there were a few details that still need to be addressed. 

First thing are pictures, for a room you should be able to get away with under 500 pictures but for full floors and houses, you will need around 1500-2000 per floor depending on the house. Now this is where RAM becomes a factor if you want to align them all at once. (And have a good amount of features for the calculations)

** You can run the alignment multiple times to refine the alignment, this is necessary for these types of projects **

For best results you want to use the following settings:

Image overlap = low (Wish mentions why in other posts, but the software will then use the whole image for alignment points)

Max Features per image = 80000 - 120000 (This is important if you are using low in image overlap but this has a big impact on RAM requirements, 32GB of RAM will be required if you are using 80k and have 2000 images, please use that as your calculation, if you have 500 images you should be fine with 120k and have 16GB RAM)

Detector sensitivity = High

Next dont try to match components first

Run the alignment. 

This should give you the best result without re-matching anything yet.

If you find you have too many components you can now try “Merge Components Only”, First without any control points and if you do not have success start adding control points to line up your components then retry the alignment with “Merge Components Only” again. 

That should give you the best alignment with the images you have. If you still are not satisfied you may need to take more pictures.

Hope this helps anyone who is working on interior models! 

 Here is an example of two floors aligned: 

Next steps for me are running a high detail reconstruction and hopefully working on the workflow process for unwraping, texture, exporting, reconstructing in something like Modo/Maya/meshlab, reimporting a low-poly mesh, retexture and finalize a model for a game engine like unreal. If anyone has a good process here please let me know. Thx!

2

Hi Mike,

thanks for sharing. I work on large VR environments (both exteriors and interiors) for Unity. I generally deal with large assets as the tolerance for both polycount and texture count is fairly high when developing for Desktop/Laptop. Generally a model will be between 1 to 5 million polygons, with around 15 to 50 8k texture maps. 

I usually reconstruct in Normal Detail using the default settings. For finer detail reduce the decimation factor to 0.5 or lower.  

The built in Simplify tool is a good start, although in my experience will generate holes. Plus, you have no control over the distribution of the polygons (dictated by your tie-points and initial reconstruction). I find that a combination of Autodesk Recap Photo, and ZBrush will produce the best low poly meshes. Recap Photo is phenomenal for filling holes, and is also great at re-baking your textures (Useful when you want to change texture count without re-texturing assets in RC). Zbrush is great for decimating your mesh and hitting your target polycount, and you can customize density using masks. 

I use Autodesk Maya to manage the various models and sub-models that comprise my final environments. When touch ups to the textures are necessary I go with Photoshop or Mudbox (using UDIM to paint directly in 3D). Finish by packing everything into an .fbx file that goes straight into Unity. 

Hope this is helpful. If you want more details just ask! 
Cheers

3

Excellent thank you! 

4

hello Mike,

thanks for sharing, I’m sure lots of RealityCapture users will find this topic very valuable, since scanning interiors is quite a challenge

here you can see the winner of our competition from the summer 2017: https://blog.sketchfab.com/sketchfab-3d-scanning-challenge-interiors/

5

Hi Mike,

thanks for brining my attention back to the image overlap! I set it to High since I though that achieves exactly what you say Low does. So what’s the difference to High then in your opinion?

6

Hi Gotz,

I was using a post that wishmaker had 2 years ago:

He stated

low-> it try use whole img area = when have small overlap 

high-> it will use “central” part

Wishgranter

  • 2 years ago

 

Hi Götz 

low-> it try use whole img area = when have small overlap 
medium-> default settings
high-> it will use “central” part ( best optical quality) of img = high overlap 

for the speed “degradation” is minimal ( few % ) so can use it freely 

But if set LOW try set in alignment higher numbers for detected points and preselector as get larger area used…

 

 

From my tests, setting the overlap to high makes alignment almost impossible, or you would need a lot more pictures and control points and most of the time would not be necessary. I am already going to be remodeling in a program like Maya/modo/Zbrush for the extra detail and 1000’s of more pictures would really not give me what I am looking as I need a lower poly model when moving this into VR or a type of game engine. 

7

Hi Mike,

I was mainly interested in your experience!  :slight_smile:

Interesting though, my last projects were all on high and I still get 99-100% of all images aligned.

I do take many though, which could explain that.

Also, accuracy is very important in my case so that’s good for me.

It would explain a few problems that I had with “outlying” areas - need to test that with medium setting and see if it gets better.

Thanks again for the insight!

8

I am 3000km away from where I took my photos :slight_smile: which made me improvise!. There was lots of rooms which matched up but the link between them was very difficult. Normally 1 room I could align pretty easy on any setting but it was the combination of multiple rooms which were lacking and having RC look at the whole image while increasing max features provided the best results.

9

 Thanks ValerioPaolux!

Honestly I am creating these environments on my own for just personal projects so its a bit difficult to justify the high prices of some of those software. Especially multiple types! 

What would be your recommendation for just one that you could clean up your mesh in for a game asset and re-texture?

I am Trialling Modo right now and am a bit stuck on how re-texturing works so im not getting optimal results. (Here is my post on trying to nail down that workflow… https://support.capturingreality.com/hc/en-us/community/posts/360000190092 )

Thanks!

 

10

Hi Mike,

I believe all the software you need to reach your goal is Reality Capture. Any additional tool can add flexibility but is not required. If I were to choose a single additional tool it would probably be Recap photo, and I would only use it to fill holes.

I noticed from your screenshot that you have 84 control points in your scene. That is a LOT. Interiors are generally much easier to align than exteriors and generally don’t require as many control points. It all comes down to how you move through the environment when shooting. Make sure to choose a path that is as continuous as possible when moving from one room to the next. Avoid jumping from place to place. Also pre-processing your images always helps. 

Here is my basic workflow to generate low-poly assets for Unity: 

  • Align images into single component 
  • Test reconstruction on critical areas. Reconstruct and Texture small areas to verify that the alignment is sound. Do this before reconstructing the entire environment to save time down the line.
  • Reconstruct in Normal Detail 
  • Filter the multi-million poly mesh. Remove extra geometry like the water tight box generated by default 
  • Simplify to 40k. This is the render upper limit cutoff in RC. This will allow you to see your mesh in Solid or Sweet mode to ensure everything looks good.
  • If your mesh if very noisy you can run a Smooth 
  • Simplify to desired polycount (depending on specs, generally few thousands to few million polys) 
  • Unwrap: i find that “fixed texel size” mode yields the best texture utilization results. I use this mode to unwrap my models. Tweak the texel size value to try and meet your ideal texture count (5-50 8k). This takes a bit of practice to get right. If you use the “Optimal Texel size” value as your “texel size” you will technically use 100% of your photos. This will likely result in more texture maps than you want. Work your way down (bigger texel size) to meet your desired texture count. (I use 8k)
  • Texture your model. I find photo-consistency mode to work best.
  • At this point you have a Unity-ready asset with correct poly and texture count. You can use Modo or just about any 3D software before texturing to improve different areas of your models such as quality of the geometry, UV utilization, textures, and such, but doing so requires a good understanding of traditional 3D. 
  • IF you do clean up your mesh outside of RC, I still recommend creating UVs and re-texturing your assets inside RC. If you want your models to look good they are going to need multiple 8k texture maps. RC does a great job at unwrapping & texturing. Dealing with UVs in a traditional way (Modo, Maya)can be extremely time consuming/challenging. Let RC do the heavy lifting for you. 

Happy to offer more workflow-related clarifications.
Cheers,

Valerio

Here is my latest file. 2655 images. 12 control points. 

11

Hi Mike,

I think MeshLab is probably the best suited for you then. It’s price to gain ratio is phenomenal!  :wink: I think there is a decimation brush included now, meaning that you can customize the areas. It’s a bit awquard to use though because it still doesn’t recognize edges. My only problem with it is that it has performance limits that are easily overwhelmed by RCs High Reconstructions.

Other than that, from my trials a while ago Last-letter-of-the-alphabet-Whisk (synonym) is the best you can get for it’s intelligent decemation master. It’s price is not nothing, but still within affordable range for a hobbyist I would think, Saying that, I haven’t afforded it myself yet and I am a professional…  :wink:

I thought that the program mentioned by Valerio from a well known CAD developer also has some kind of limited free version? Because a good hole-filler is essential due to the holes that Valerio mentioned.

 

Hi Valerio,

just out of curiosity, why do you use decimation 0.5 in normal reconstruction?? Isn’t that just the same as high detail reconstruction? Does it really change anything? Because I can’t see how there could be more vertices than the depth maps provide, which would be implied by a factor lower than 1.

Another way of getting better-to-handle meshes would be to set the minimal distance of vertices - I haven’t tried it myself yet though. Does anyone know wheter this is something that interferes with the calculation before the final mesh or is it just a more elaborate simplification which would then again create holes?

12

Hey Valerio,

this is weird, I can only see your last post now, 3 hours after my last one!

Anyway, I think this is a very good summary! I agree with you entirely (apart from the statement that interiors are easier than exteriors, but that is very much a personal matter and also one of experience I think). Just a shame one can’t fix the holes inside RC, but that might change in the future. The only reason for an external unwrap is if people need to work with the texture afterwards, as in changing it, enhancing it artistically etc. Because for that one needs an unwrap that has as few parts as possible where most of the triangles are attached as they are on the mesh. This is the only thing that RC cannot provide at this point because RCs unwrap looks like a section of a sanbox with artefacts in it… :slight_smile: Of course that also depends on the object - the more and larger plain faces it has, the less of an issue is that…

When you say you simplify to 40k Tris, do you really mean that or rather 40m? Also, the amount that RC can handle depends on the memory available, so there can be significant differences…

13

Fabulous wisdom - thanks Mike.

You say

“Max Features per image = 80000 - 120000 (This is important if you are using low in image overlap but this has a big impact on RAM requirements, 32GB of RAM will be required if you are using 80k and have 2000 images”

It may not be necessary to set Max Features per image (and Max Features perMpx) so high - or alternatively may need to be even higher. See

https://support.capturingreality.com/hc/en-us/community/posts/115003012052-Optimising-Alignment

 

 

14

@Götz
It looks like my posts are being delayed. I am just learning the ways of the forum, and how to phrase my posts wisely.
40 millions is indeed what I meant for Solid Mesh display.

The Texture touch-ups are a key step as no environment is ever captured in its entirety (occlusion).I often have to remove objects, fix floors, or simply fill in areas with missing textures or slight misalignment, but still stick to RC and avoid manual unwraps for 99% of the work.

I then export models as UDIM and paint textures in 3D instead of 2D and do not  worry about seams. For any ad-hoc work I just detach small areas of the models and unwrap separately into a single additional 8k that can be easily managed in 2D (sometimes you can simply add to an existing tile)

This of course requires a bit of planning around mesh creation and splitting environments into multiple models (eg. rooms) so as to keep a manageable per-obj texture count.

I think RC unwraps are actually quite good, and surpassed only by…(same tool mentioned earlier) with some limitations. Texture utilization averaging around 75% is quite amazing for the time required. I need dozens of maps to meet resolution requirements, and Utilization is the most important thing in the end.

More custom Unwrap options would be great though (eg. ability to paint/preserve seams or groups of faces without having to separate geometry. 

What is your workflow for dealing with UVs, and touch-ups?

15

Hi Valerio,

it’s interesting that you say the unwraps are quite good because so far people were rather critical.

But that just prooves my point (that I often make when people ask about “the best workflow”) that there are so many different workflows that require so many different features that it’s totally impossible to bake them all into one software. So where we have similar objects, we still have very different needs and consequently workflows.

I personally couldn’t worry less about the unwrap since I keep everything within RC. I am mostly interested in orthophotos for buildings. They need to be as accurate as possible since I use them for documentation. So I cannot just fiddle around with some parts and make them look nice - which I often wish I could!  :slight_smile:

The only thing I do to the finished orthos is to adjust the color slightly and sometimes touch up some areas that got brighter than the rest because I had to go in closer. All in IPS and 2D, so nothing fancy there at all!

16

Hi Mike,

You said; “Valve has an amazing guide here but there were a few details that still need to be addressed.”

Is this the guide you were referring to:

https://developer.valvesoftware.com/wiki/SteamVR/Environments/Photogrammetry

Thanks for posting!

17

After spending time playing around with RC Unwrap, I agree with ValerioPaolux that the RC unwraps are actually quite good. I also like that RC Unwrap gives complete control of pixel resolution in real world units.

18

Tha real size pixels/texels are great indeed!

If only there was a way to determine the actual distribution of sizes for complex models, like a histogram or something, it would be even better!

19

Well, for a massive interior, RC really can struggle with UV’s and I also am trying to keep texturing efficient so I dont have 50 different textures. 

Using both a low poly texture for walls and simple objects are the key to a nice finish for a VR product, my problem is how to handle UV’s in 3rd party programs then using them back in RC.

Also I have no idea what baking a mesh means from a high to low poly, Is that for the mesh it self or just the texture? ugh so much to learn, I just need a good course to take which will get me up to speed quickly.

20

“I just need a good course to take which will get me up to speed quickly”

Amen to that