Satellite data conversion for 1/3rd ARC sec data

Development Asset Creation
1

Hello all,
I have downloaded ALL of the 1/3rd Arc Second Satellite data for the main part of the United States I.E not Alaska, Hawaii or any of the outlying US Islands.

I have run a report over all the data which is in GeoTiff format ( floating point tiff - with extra tag information ) to obtain all the information that I need which is :
Minimum height below sea level ( sea level treated as 0 )
Maximum height
No data values ( literally a value that represents no data )
NaN values ( A value that represents - Not A Number )
Longitude and Latitudes for the Top Left, Top Right, Bottom Left, Bottom Right of each image.

This has told me that I do have No Data Values and what the data for No Value is for each file, that I do have NaN values. That the values over all files for below sea level height is -111.04142 , the maximum height is 4414.2256.

I have written some python code to convert these files from GeoTiff floating point values to Unsigned Integer files compatible with Unreal’s .r16 heightmap format - except that they are WAY TO LARGE. Each 1 Arc second file at 1/3rd resolution I.E 1 pixel = around 10m is 10812 x 10812 pixels before conversion take about 480 Mb each.

My next step is to write some more python to convert the data into Unreal friendly sized .R16 file chunks each of which will have a standardized file name so I can control load order via a chunk loader; however before I do this I have struck some problems with being able to visualize the data - just because it is so HUGE to make sure my algorithm for conversion of floating point values to unsigned 16 bit values for import are at the correct height and I am scratching my head with regard to it…

So now you know the whole story what I am trying to determine is:
What should I set NaN values to when outputting from the floating point to Unsigned values.
What should I set No Data values to when outputting from the floating point to Unsigned values.
How should I set the data in general, at the moment I use the following Python snippet for conversion where BaseHeight is the minimum value from my reports + 1 as a positive value I.E: 112

NB: I’ve been looking at this so long I’m going cross eyed having processed 389 Gb of Satellite data in 1120 files so some input from somebody who has tried this sort of thing would be invaluable.

Thanks all, look forward to some responses.

                if math.isnan(Array[x, y]):
                    data = 0
                elif Array[x, y] == NoDataValue:
                    data = BaseHeight
                else:
                    data = Array[x, y]

                if data != 0:
                    NewHeight = BaseHeight + data
                else:
                    NewHeight = data

                Converted = int(NewHeight)

                if Converted > 255:
                    lo = int(Converted % 256)
                    hi = int((Converted - lo) / 256)
                else:
                    lo = int(Converted)
                    hi = int(0)

                outme.write(bytes([hi]))
                outme.write(bytes([lo]))
1 Like
2

This is me replying to my own post - as it has a bug ( quite an obvious one too )

This is what the writes should be: oops.
outme.write(bytes([lo]))
outme.write(bytes([hi]))

3

Unreal is a game engine, not a geoviz engine. 32-bit floating point does not have sufficient mantissa resolution to represent continent-scale features – typically, the max size difference you’ll want to deal with between “maximum” and “minimum” points in 32-bit is about 8 km, although this can vary by a few factors of 2 (each way) depending on specifics.

Having written a full-sphere engine many years ago, the approaches are different, and you’d need to do a number of manual levels of LOD to make anything like “all of the US” work in the Unreal engine.

E g, start out with 1 sample == 1km at 10m unit size for the overview, then when zooming in enough, swap to 1 sample == 100m and 1m unit size, and then when zooming in enough from THAT, swap to 1 sample == 10m and 0.1m unit size. (which is equivalent to 1/3 arcsec.)

You will need to create streaming levels that cover about 1000 samples, and page in the right number of those levels. Also, as you switch scales, you’ll have to do something to hide the LOD pop, and change all your camera and physics offsets.

Also, because physics assumes centimeter units, you’ll need to do something about that, too.

There is a 64-bit branch on GitHub where there’s work going on to try to turn the engine 64-bit capable, which would be sufficient to represent Earth. Typically, your rendering and interactions still happen in 32-bit floating point, but they are all relative to some 64-bit frame of reference, so each object has a 64-bit “global position” but when rendered, is relative to the camera, or some other reference point, measured in 32-bit.