HiresSattaliteData (Landscape stair steping effect)

Development World Creation
,
1

Hi All,

I have been doing some testing with HiresSattaliteData from Mars. I’m using IMG files in combination with Qgis and TerreSculptor. The heightdata gets imported into Qgis and Geotiff file gets
imported into terrescluptor. When I import the 16 bit greyscale png into UE landscape I’m getting stair stepping on my Landscape. Does anybody know what causing this? The Lanscape is very noisy. What can I check?

export HM_Landscape.png from Terresculptor 256 units | 1 unit=1cm

Its 8129 x 8129 landsape.

Hope to hear from you.

cheers
decorix

Landscape_star_stepping.jpg
Landscape_star_stepping.jpg1165×621 173 KB

.

2

Hi
I use a similar workflow, Qgis, TS, UE$

How are you getting the heightmap data out of QGis ?
There is a plug in to output heightmap
QGIS Python Plugins Repository

Rendering out an image from Qgis will not produce a proper 16 bit heightmap

Paul G

3

Yes it does, it outputs whatever you tell it to output in whatever format you tell it to output.
save raster as, and swap to rendered image.

As far as OP issue goes. that’s normal.
The data from the image is 1 meter per pixel and 1 bit per meter of height so terracing is very normal unless the lidar readings are around 1m as well.

Get Nip2 and set up a script to radial blur the image and export tiles all at once.

4

Hello Paul,

Thanks for you kind help and information regarding the plugin. I have send you PM btw.

I just installed the plugin. not tested it yet, I’'m not sure what settings I need to define for exporting to unreal.
Is there any info in the IMG file I need to set in the height_exporter dialog?

Thanks

Decorix

5

False information,

Without the plug in Qgis just renders an image
Nothing to do with the height data contained in the Raster layer

Suggest you try the heightmap export plug in before decrying the solution declaring that terracing is “normal”
Terracing is not normal if you have hi res DEM data
Paul G

6

Corey
in the heightmap export, put in the model dimensions, and as your data is extremely large you may have to put in a scale factor
this will create entries in the output sizes and enable the export button.
It exports a 16 bit PNG
If req you can then import into Photoshop and split into tiles, (PS does have a slight degrading effect)
These tiles will go straight into UE4 , TS is then optional
Alternatively your exported heightmap may go into TS and use the tile creator to export a set of tiles

Paul

7

Terracing is normal on most datum given the value ranges avaliable. This isn’t a lidar point cloud we are dealing with, but range distance readings of Mars. I doubt the datum is sub 1m.

next, if you just adjust the value range you can manually export a png raster that just works no matter what plugin you use. Despite the plugin being useful or not.

The height data contained in the raster layer is what is being rendered. Black to white 0 to 65536 range.
You can actually define your own range by adapting the layer as well. That becomes important since when you bring it to unreal you need to set the Z scale correctly.
Its even more important when you need to bump up the 0 in the maps to 6553/2. Normally earth 20m datum have sea-level at 0 val. You would have to shift that up to where unreal puts sea level to get proper heights.

Additionally. The map showed by OP as a single png16. Is 428,881m or 428km x 387km at a 1:1 size in px per meter thats 428881 x 387000 = 162,117,018,000 bits which is around 20GB
for a single png16 that won’t really fly. You can save it in geotiff to preserve the data. I have a 40gb one saved as such.

At OP, you need to cut the tiles up just like you would to bring it into the engine in world composition.

nip2 is the best way to do just that, but you have to save an initial image that’s the least compressed possible with the correct 16bit values of your liking.
not all formats support a 20GB file, so you have to pick one that does.

8

btw, I never use it, because I stick to nip2 which does wonders. but here’s a python script that works within qgis for tiles. you need to adjust it for your needs ofc.


import os
import gdal
import subprocess

source_path = 'PNG File'
tiles_folder = 'DESTINATION FOLDER'

tile_size_x = 8129
tile_size_y = 8129

tile_prefix = "tile"
tile_x = 0
tile_y = 0

ds = gdal.Open(source_path)
band = ds.GetRasterBand(1)
xsize = band.XSize - 1
ysize = band.YSize - 1

min_height = 0
max_height = 65535

for i in range(0, xsize, tile_size_x - 1):
for j in range(0, ysize, tile_size_y - 1):
format = "-ot UInt16 -of PNG -scale " + str(min_height) + " " + str(max_height) + " 0 65535"
cutting_frame = "-srcwin " + str(i) + " " + str(j) + " " + str(tile_size_x) + " " + str(tile_size_y)
output_path = tiles_folder + tile_prefix + "_x" + str(tile_x) + "_y" + str(tile_y) + ".png"
full_command = "gdal_translate " + format + " " + cutting_frame + " " + source_path + " " + output_path
#os.system(full_command)
# version with subprocess module
args = 'gdal_translate', '-ot', 'UInt16', '-of', 'PNG', '-scale', str(min_height), str(max_height), '0', '65535', '-srcwin', str(i), str(j), str(tile_size_x), str(tile_size_y), source_path , output_path]
subprocess.check_call(args)
tile_y = tile_y + 1
tile_x = tile_x + 1
tile_y = 0
9

Thanks for your feedback Paul. Using the Height exporter it looks much better in UE4. How do I calculate model dimensions? I’m not sure what numbers I should use. Is this how large the actual terrain is? I need to punch in some numbers otherwise I can’t export. What do you recommend?

10

Hello MostHost. Thanks for your help and detailed information. Can you perhaps explain what tweaks you normally do when editing grayscale values in Qgis.
What are these evualtion numers -2678.2 and -2389.86. I know that I need to change the Z value in Unreal.

11

Hi Corey,
Add in the actual model dimensions, (listed in dialog as map dimensions) probably with a scale factor as your image is huge
then the image dimensions fill in automatically.

Not sure that the IMG is suitable for terrain it covers a huge area and although the model size is approaching square, the preview is very rectangular.
The altitude range is also very small in the model details
Import projection could have and effect Qgis usually use earth based grid projections

Paul

12

The most common operation to get a raster file to be displaying in a heightmap range is to use the raster calculator and do
(“layer” > 0) * “layer” + 32768

Layer isn’t a thing, you need to actually use your merged layer name from qgis. the calculator display them at the top left. just double-click the appropriate one and it will be put into the equation below.

Running that code effectively converts any value above 0 in the raster to the height value that unreal puts the sea level at + whatever measure the data in the raster has.

This is half the picture. To get the raster to display as a PNG16 you need to scale the band values from 0 to 65535.

Exporting the resulting raster as just about anything should produce a working file with the appropriate value ranges.

13

Couple added things / let’s call them new discoveries brought about from OP’s post and the erroneous assertion that “a plugin is all you need” which is definitely wrong.

When you save a layer in Qgis - Right click > Export > Save As, you have 2 options. Raw Data and Rendered Image.
Rendered Image always saves in 8bit (and I haven’t been able to force more).
Raw Data always saves in 32bit.

For processing your image into a PNG16 you need this workflow:

Get tiles, merge them into a single layer (Raster > Misc > Merge)
Run Gdal_translate. (Raster > Conversion > Translate in current version) add -ot UInt16 in the “Additional command-line parameters”.
Set the file to save to, avoid temp if you are dealing with 100km^2 plus.
](filedata/fetch?id=1861831&d=1613149978)

This saves out a 32bit tiff with the correct data.

Run the calculator with
(“layer” > 0) * “layer” + 32768

Save that to a file as it’s your final raster from Qgis with the proper data set.

You can now acces the raster properties and change the gray band in a 0 to 65535 range.
export and save as raw to finalize.

(Note: the Z value in unreal now needs to allow for a range of ± 65535 meters since the heightmap was setup to have 1m per bit. The Magic z number scale value for that is 12,799.8046875.
obviously you loose some resolution doing this, as the terrain will now lift up or dip down in 1 meter increments).

Now you can run Nip2

And convert from whatever geotiff was saved (in raw) to PNG16.

The resulting file may be impossible for most programs to handle. Qgis, Pyton, Img magic, and any other command line utility doesn’t care what the size is since it has not real need to render it (except if you make the mistake of activating it in Qgis or something).
You can use the script I already shared to obtain Unreal Ready tiles. (unfortunately Nip2 doesn’t name them right, so yes you can cut tiles, but then you have to rename them all…)

14

Thanks so much again for you detailed info. I’m a new Qgis user. So I really need to take all the information in :slight_smile:

What do you mean with (Get tiles, merge them into a single layer (Raster > Misc > Merge))

Does this create tiles? If I understand correctly this is to be set up with Worldcomposition? and this I know requires tiles. Can I skip the NIP2 part if want to use other sofware.
So the whole purpose is to convert data into 0-65535 range.
And it needs several raster passes to do this?

How does the final raster look like would it have more contrast then the Renderd version.

15

It all depends on the data set and what degree of accuracy you are going for in engine.

assuming 100% accuracy and 0 visual fidelity the stair stepping increase using the raster calculator I already described will suffice and be far more accurate than anything else I have tried.

If you are going for visual fidelity having ±65535 meters on a landscape is obviously bad, as it will show clear stepping between the shifts.

As far as how the raster looks visually, insignificant as it is, a soft gray to full white.
Ideally, if the underwater portion offers some terrain this part will range from 0 black to the soft gray of the sea level (no map I have found for lidar readings contains underwater data, I suppose data can actually be found for it to patch it together perhaps. Usually I just sculpt it in engine).

To answer your other question, when you find data online its usually comprised of several tiles.
each tile has it’s own min/max range, causing visible seams.
Running the merge operator assembles a complete image with the corrected range so that no seams exist.

Theoretically, the best possible outcome would be to change the raster so that the highest value of it is 65535, and the lowest value (sea level) is 32786.

​​I’m working on the raster calculator equation to do just that, so that one can have a decent visual along with a generally accurate height by just doing the proper Z scale math.

16

It is not erroneous,

The Heightmap plug in exports 16 bit PNG from whatever Raster Elevation data you have in the QGis project

Why do you insist on putting down a simple solution to the OP’s problem when you by your own admission have not even tried the heightmap exporter ? and then then produce the most convoluted half explained series of bumf that no one has the chance of following.

Post Reported

Paul G

17

Because it’s a completely wrong, not to mention counterproductive approach.
The export plugin doesn’t do anything useful to begin with. IT will never work on a large data set. AND to top it off, it’s just a basic gdal script that mostly fails.

@OP
You can actually run the following queries to get your map into the proper range - for your mars use case you can skip the shifting step.

  1. Convert to UInt16 with gdal_translate as described before. the file is now in the 0 to 65535 range.

  2. Shift the raster values up by it’s max value. You can find the value in the details Or, more convinienly if you didn’t modify the raster grayscale band, right in the Layers View.

https://forums.unrealengine.com/core/image/gif;base64
​
To execute this step you just run this inside raster calculator:
[FONT=Courier New](“layer@1” > 0) * “layer@1” + MAXVAL
where layer is the raster you need to manipulate, and MAXVAL is the value you obtained from the file itself. (currently MAX funtion is broken, in the future maybe MAX(“layer@1”) will be a valid way to skip having to find the value.

The resulting raster is now in the “correct” range we need to have sea level at 0 in Unreal (half the overall height).

  1. Run the claculator again to scale up the raster from it’s current min/max to 0 to 65535:
    [FONT=Courier New]“layer@1”/ (MAXVAL*2) * 65535

This puts the data in the file into the correct value range for export.

  1. To be safe, run step 1 on the final file again. the raster calculator has a tendency to shift back to 32bit
    Technically the product of this step is a ready to go 16 bit file with proper range. You just need to convert it from Tiff to PNG

  2. right click, export raw in 32bit (set columns to 1 for 1:1 size resulting in a very large file).
    BEWARE the stretching of the data. if the columns are set to 1:1 from a file with less than 1m data you will get stair stepping

  3. run the tile script (page one of this thread), Or, if your final image is already sized for the engine, convert it from raw tif to PNG16 using Nip2 or any other capable program.

You should thereafter be able to take the file to the engine with minimal stepping (depending on data), and proper sizing.
Just use the MAX height in CM * 0.001953125 as defined by the engine docs.

23

Cleaned up the flaming in the thread. Stay on topic or take it to DM’s please.

Please don’t do this. The “Report” feature is not there for reporting posts you may disagree with, it’s for reporting spam and general poor behaviour.