Texture crisis -- a plea for workarounds

Development Asset Creation
,
1

And yes, I do mean crisis.

I’m completely blocked by RC’s newfound propensity to halt texturing with an error just after unwrapping with either the ‘new’ or ‘legacy’ algorithms.

I had no trouble texturing a simplified mesh from my ~4000 camera project using RC CLI.

Now, after re-aligning the same data set, RC fails to texture a similarly simplified model with this error:

The first time I encountered this ‘sanity check’ message, it was when I had exported an .obj and re-imported to RC.

But now I’m seeing it in a mesh generated by RC itself, with the canonical pipeline and hard-reset RC setttings:

  • import images
  • align
  • reconstruction
  • simplification
  • texture

I can still texture small files, but not the larger project I need to finish.

Is there any kind soul who can suggest a workaround of any kind?  I fear I have to completely abandon RC otherwise as I cannot use a model without texture.

I’ve tried the following:

  • Resetting RC settings:  same error
  • Cleared cache and system temp:  same error
  • Created new reconstructions, large and small:  same error
  • Exported to Maya, ZBrush, GSI, MeshLab to edit and re-import:  same error
  • Built proxy geometry from scratch using the RC output mesh as a template:  same error
  • Tested smaller subsets of the ~4,000 camera data set:  texturing works in small projects

I’m at the end of my tether and any support or tips would be very much appreciated.

2

My plea above is for any help getting a large scene to compute textures without the ‘sanity check’ error noted above.

I don’t want to draw attention from that, but below is an addendum as I’ve tried to understand the circumstances of the problem – which entails it’s own problems:

Since texturing mostly works with my small scenes (<100 images) and not with my large scene (~4000 images) I’d like to cut the large scene into pieces to see at what point things stop working.  We can’t delete images/cameras, so the only way to test that I can see involves stepping through thousands of cameras in the file by hand, one by one, to turn them off for a given reconstruction. 

We can export cameras and images as .fbx, for instance, but then I’m stuck trying to group thousands of cameras in the file by hand and we can’t import .fbx back to RC so that also seems not to work.

3

Hi Kevin - rather than one-at-a-time on the pictures you can shift-click bunches of them and disable them for texturing (the inputs box applies to all the selected images)

At least that way you could play with the numbers to use anyway. 

Is there a chance this is related to the licence limit of 2500 photos for some classes of licences?

Good luck on the test… Jennifer 

4

Hey Jennifer,

2500 should be ruled out since he has the CLI, but you never know - would be easy enough to test.

 

Hey Kevin,

do you know the grouping menu that is activated by clicking on +Images in the 1D menu? Makes grouping possible in one click, provided you don’t have any special requirements. Why do you want to export the cameras? Just to be sure? Have you tried using export Registration and importing that as a component into a new file?

One more idea, but I guess you already came across this: it seems like intersecting surfaces due to slight misalignments can cause RC to react erratically and in your intricate project I can see dozens of places for something like that to happen…

5

Another silly idea - have you tried texturing in a third party software? With the camera export it should be rather straightforward, even MeshLab does that. Although the amount of images could be a challenge…

6

Hello and thanks Jennifer and Götz!

Your mutual points about grouping are good – but since the camera list is not currently grouped, it would take thousands of manual clicks to group them appropriately now.

Jennifer, as Götz noted, I’m using the CLI version so theoretically the number of cameras/images is fine.

Götz, I understand your fear that RC could be acting strangely with intersecting geometry, but I’ve successfully textured this data set in the past weeks, with no problems.  The only difference is that I’ve subsequently managed slightly better alignment.  There is nothing novel in the scene after the last alignment round – identical poly count, identical input images, only slightly better alignment.  So I don’t see why RC is now giving texture errors for a fundamentally identical reconstruction. 

I could throw away my hard-won alignment (hard won because alignment itself crashes RC after hours of computation in this project, as you’ve seen in a separate thread) to get back to the previous textured version, but that seems silly.

Instead, I hope is that somebody has figured out when, and why, this particular ‘texture sanity check’ error arises.

A hint:  if I strip out geometry to a minimal number of triangles, RC completes the texture.  I don’t know why.

To answer Götz’s other question and suggestion:  I have thought of exporting the cameras to Maya to bake a texture from their aggregate projection, but I see in this forum that people seem to have problems getting thousands of cameras to import correctly from RC–>Maya, so I haven’t been enthusiastic about that.  Better and simpler I think to keep this in RC, where a core function is texturing.  If people think that ~4K cameras is viable in Mudbox/Maya/et cetera, I’ll give that a go but again prefer to stay in RC as the mesh reconstruction is native to RC and it should be able to texture its own simplified mesh.

7

Oh sorry - this wasn’t a suggestion about making a group of them…  Instead just group-select a bunch with shift click, then set the selected cameras to not be used for texturing (just to see if fewer cameras gets you over the line).

Then select all the cameras and return them to service.  No playing with the group id’s etc -  just  bulk selects.  you could even lasso select in 3d to disable them by region… 

8

Hi Kevin,

I’m always reluctant to point things out to you because you are clearly playing in a different league.  :slight_smile:

But I don’t understand your point about grouping - did you not mean exif grouping? That would be one click for all cameras grouped. You can also select many, like Jennifer has pointed out, and change whatever value you want for all of them.

About the 2500 - it might be worth a try, maybe there is some little ■■■■■ in the code somewhere.

I see your point about the hard won alignment. Although your experiments prove (in my view) that it is somehow tied to the mesh geometry, since it seems to disappear when you simplify it to a high degree. So the intersecting faces do seem like a possibility for the source of the error. Have you tried cutting random parts of the mesh away to narrow the spot down? Maybe downscaling images for texturing helps reduce calculation times for this.

Sometimes it can be faster to talk about such things in person. If you think I might be able to get you any further, feel free to call me - you should be able to find my details in google with my name…

9

Thanks again Jennifer, that’s a good suggestion, I don’t know why I was making it more complicated in my mind.  :wink:

I group selected all the cameras and turned their texture contribution off, then picked just a few cameras to turn on as a test.  That worked – or at least got past the ‘sanity check’ stage, like my test where I arbitrarily removed 99% of the triangles in the mesh.

Unfortunately, even with just (3) images and an 8000 triangle count mesh, RC crashed 50% of the way to completion when it ran out the 128GB of memory available!

I can’t begin to wrap my mind around how RC can spend 128GB of RAM to texture an 8000 triangle model with three photos…I must be doing something odd, but I have checked again and am using stock (reset) settings for RC globally.

All of which suggests that there is a camera/geometry line here, and makes me wonder how I was able to texture models as large before!

10

Thanks also Götz, especially for your very kind offer to try to sort through this on the phone.  I hate to abuse your kindness, but given my desperation with this particular project I won’t rule it out!

Let me first do some more homework with re-projecting the corrected images and RC camera positions in other software, to see what kind of reprojection error there might be and whether dealing with thousands of 24MP images is tractable for VFX software pipelines.  While those kinds of demands are nothing special in the MVS community, VFX software is still built around expectations of hand-built geometry and traditionally uv texture atlases, not thousands of images and very large models.

11

Hi Kevin, no worries - I offered!  :-)  The worst that can happen is that we both wasted a few minutes on the phone…

My bets are still on the geometry. Better numbers in alignment does not necessarily mean “better” geometry in my experience. So given your project I could imagine some tiny metal object in the background somewhere that causes some inconsistent geometry.

12

Hi Kevin Cain 

What settings are you using for UNWRAP ? can you make screenshot just of the  UNWRAP settings  used there ? 

13

Hello, @Wishgranter,

Here are my unwrap settings for this project:

The settings for the ‘Unwrap’ tool in Reconstruction:Tools match the above.

I’ve tried both the new and ‘legacy’ simplification and unwrapping.  The new simplification hangs when converting the ~750M triangle source to 3M triangle target, so I’ve defaulted to ‘legacy’ in both simplification and unwrapping.

14

I don’t know if this relates to textures failing with the ‘sanity check’ error above, but I notice that RC is again turning off groups of aligned cameras by itself, which I reported earlier and was assigned bug 6983 here.

 Here’s an example of a block that was turned off:

 

15

When I explicitly call the unwrap tool before texturing, I sometimes get an unnamed error banner shown below:

As you can see in the upper right, the last console output is ‘Texturing model failed…’ with no further details.

I’ve seen this with settings for [1 tile at 8K], and [1 tile at 512].

16

Here’s a summary of the sanity check error in this thread and what I’ve tried – all comments welcome!

 

The ‘sanity check’ error and steps to reproduce:

 

Before the problems in this thread, I successfully textured a simplified 3M triangle model from ~4,000 aligned source images – let’s call this reconstruction ‘A’, via these usual steps:

  • import images, CPs from .csv (previously exported by RC)
  • align (iterative)
  • reconstruction at normal detail with hand placed bounding box
  • simplification to [3M triangles, 1 part] from above reconstruction [751.5M, 500 parts]
  • texture [24 tiles x 16K]

So far, so good.  Next, reconstruction ‘B’ shares ~3,900 of the same source images as reconstruction ‘A’.  The difference, as noted in this thread, is that RC added ~100 cameras to the prior ~4,000 camera alignment between reconstruction ‘A’ and ‘B’.

 

While I had no problem texturing reconstruction ‘A’ up to [24 tiles x 16K], reconstruction ‘B’ fails even with [1 tile x 8K].  If I explicitly call unwrapping, it consistently gives the unwrapping error shown (at 512 to 8K) but succeeds at 16K.

 

Even when uv unwapping succeeds, during texturing RC gives the ‘Texturing stage 01 sanity check’ error that is the subject of this thread.

 

What works and potential clues:

 

If I manually turn off texturing for all but a few cameras in the ‘Camera poses’, RC will complete the texture – but of course, only with the contribution of those few cameras.

 

After many trials I’ve increased the number of cameras with texturing on to a maximum of about 1/4 of the cameras in the scene, ~1000.  Any more than this again gives the ‘sanity check’ error.

I’ve manually winnowed the camera list to see if individual cameras are causing the error, but so far it seems that it doesn’t matter which cameras are set to participate in texturing, just the total number of cameras set to participate.

 

After texturing fails, RC will sometimes turn off groups of cameras in the ‘Camera poses’, as shown above.  This is not consistent.

17

have you tried more than a max of 10 textures?

for that many photos, i’d be setting the max to 32 or 64. it will still only use as many as it needs.

have you also tried photo consistency? it slower but it may help. 

i would also have a go at 16k textures too. they will be easier to resize smaller less textures anyway.

I would also just do unwrap stage until you have that looking like its all working ok.

 

there is probably a small part somewhere in the mesh that’s a bit dodgy, it might be worth exporting mesh to see if you can see any parts that could be causing the issue, and then maybe make a new reconstruction region to build a new mesh that dosen’t include it.

 

 

18

Following Jennifer Cross’ suggestion, I manually turned off texturing in groups, and by walking through the whole set of images in this way was able to isolate a few images that were causing the texturing to crash after unwrapping. 

The lesson (for me, at least) is that images where RC finds very few correspondences can cause crashing during texturing.

This gave me part of the answer – at least I have been able to get a rendered result in most cases – and now I’m stuck with a new problem, strange texture results.  But that is a new topic for another thread.

Thanks @chris, I agree that it’s best to unwrap first, then texture.  Your other suggestions were also well taken, though they didn’t relate the crashes in my case.  Thanks again!

19

Hey Kevin,

that’s a very helpful observation, thank you!

By correspondences do you mean Tie Points or “partner” images? Is there an easy way to determine the latter?

20

Tie points are the crucial numbers to watch for this in the case of this crash – an image with less than 10 tie points is able to reliably cause crashes in my experience.

However this is connected to partner images, in that user-supplied control points establish image neighbors.

If I had to guess, I would say that when RC cannot reconcile user-supplied control points during bundle adjustment, it’s left with erroneous points and therefore erroneous tie points.  The default weighting value of 10 isn’t enough to drag the tie points from SIFT too far out of alignment, since they are much more numerous, even for relatively featureless images. 

However, in the ‘right’ cases. these points are enough to cause RC to establish a few tie points with images whose adjacency doesn’t match what it expects from bundle adjustment – and this disparity leads to the crash.  That is, RC should know better but can’t downweight the user control points – because they are correct (see below), but in locally aligned groups that don’t manage to be reconciled into a global alignment.  In my guess, only, as RC is not open source.  :wink:

I say this because my observation is that in complex scenes with hundreds of control points, even with high quality points (that is, placed < 1 pixel in reprojection) RC sometimes can’t find a global alignment that fits all points.  In this case, it reports high errors for control points we can objectively see are well correlated.

In these cases, for whatever reasons, RC alignment stops in a local minima, stopping short of a BA that fits all points.  This is to some extent unavoidable as in incremental sequential alignment, large scenes accumulate error.  A global SfM approach would be better at closing the loop in these cases, but more brittle and likely would incorporate fewer images of the whole set.

It seems RC prefers to try to make use of all possible images but is then unable to square the alignment error vis-a-vis user control points in complicated scenes.

If anyone has any experiences along these lines I’d love to hear.