Tiled worlds

Development World Creation
,
1

Hey everyone. I’ve been looking around for more information on tiled landscapes, but the questions I have haven’t really been answered anywhere, so I thought I’d open up a discussion where i could post my results or get some hints.

My goal is to create a very big, tiled world with a reasonably good polygon density. Some recent improvements like Virtual Texturing are nifty and we’re going to make full use of them in an attempt to make some persistent runtime texture changes to the landscape.

The challenges are:

  1. Generating and exporting
  2. Importing into UE
  3. Effectively texturing the landscape using generated maps like flow and normals
  4. Managing it in the editor and in-game.

Generating and exporting
Generating and exporting is pretty easy. If you’re on World Machine Pro you can just make a tiled world build. The first thing I established is that it creates the tiles from within the extents of the previewed landscape, so what you see is what you’re going to get. That confusing tiled world view inside WM means nothing.

If you don’t have World Machine Pro I also wrote a photoshop script to slice up an existing single image. If you’re exporting as .r16 format (and you should be) then you want to go to “Open As…” and in the dropdown select Photoshop Raw. From there set it to 1 channel, 16 bits, IBM-PC byte order. Everything else you can leave as-is. Then run the script (it’s in another thread, I’ll link it later).

Ultimately I chose to generate a 16kx16k landscape divided into 64x64 253x253 pixel tiles. I assume this is around 20km squared. This is about one polygon for every 120 units of world space, which isn’t actually too bad for detail scale, but we may want to go even more detailed if everything works out. All I did for the landscape was an Advanced Perlin Noise so it’d generate and export quickly. That still took five minutes and it generated over 3000 tiles. Then I had to do it again because I hadn’t told it where I wanted it to save them.

Importing
Importing into UE isn’t so bad either, but it’ll really pay to have your materials and weight layer files sorted out in advance. Reimporting really big worlds is tedious.

Texturing
I’ll come back to this one after I finish with importing.

Managing the landscape
This is where most of my questions lie. How many tiles should we try to load at once? How many tiles should we have at all? Can we display imposters in the place of levels not streamed in yet My earlier attempts resulted in a lot of sitting around staring at UE while things either loaded or messed up. I want things to go wrong so I can solve them, so rather than start with a small tiled world we’re going to start with a colossal one.

It turns out all the answers are right here: Let's make an MMO in UE4 - Part 8 (HUGE Fast Landscapes) - YouTube

You can easily set up level streaming distances and LOD proxies for the streaming level.

2

Well while this may not matter or apply to you world machine pro isnt the only kid on the block anymore, and while it may have more features,L3DT pro is now free.
This is maily for others or yourself as it may be, to make everyone financially challenged on any level, know options exist that level the playing field.

I’m having giant headaches with ‘world composition’ myself, less as of now, but I still have yet to fully grasp , level streaming, and there are several ways to do it.
You should load in as many as you need to fulfill your world ideas, figuring out the size you need ( ue4 terrain size guidelines: Landscape Technical Guide | Unreal Engine Documentation .

Judge from there accordingly. So far I have ( chose ‘wrong’ sizes b4 I knew better , and its OK for now) 4 tiled levels @ 5101x5101, & my worse headache is GETTING tile 1 as EX, to be visible as I’m near tile2, but not working atm,I suspect part of the answer may be in level LOD, but haven’t delved deep into that yet.

The idea: It horribly blows reaism when tiles ‘load’ in and out,I can hide it to some degree based on my current know how, via trees, whatnot,but there is no doubt, maybe with LOD, a far far better way as some levels may be simple by design, so that would not work.

Cheers

3

This thread is more about what happens in UE after your world is generated, I just put the other stuff in because it’s where most people start.

The main thing I’m noticing about streaming tiled worlds is that there’s no in-editor utility for automatically loading them. That makes perfect sense, but what happens during runtime? What do we see before a level is streamed in? Do we need to write logic to stream stuff in? Nothing? Is it possible to generate an imposter of the level to show before it’s loaded? Obviously I’m missing out on a couple of tutorials here so I’ll go find them, but the LOD stuff is the thing I can’t find anything on. At the very least I want to see distant landscape tiles all the time.

4

At the very least I want to see distant landscape tiles all the time.< atm thats what I want, but having seen a few tuts on youtube from epic staff (yr ago orso),either it can’t be done easily,or they left it out.

I even tried disabling streaming so its not supposed to have any,put the 3 tile in to hopefully have them all included and loading always but nadda.

I do know one thing or so I’ve heard, the world comp code is no longer being maintained…dunno why…and supposedly not-- think that was mentioned on youtube vid by epic staffer,anyway I do wonder if LOD,unsure also how they work,may be key to some level of realism.

Depending on size of world,seeing them all the time might work for you( in EXE), who knows whom else, at very least it will prob future proof project LOL

5

I suggest you have a look at this tutorial serieshttps://youtube.com/watch?v=YbnGVrpIGEoAlso for @neighborlee this specific video for loading/streaming in.

RE:
How many tiles should we try to load at once?
All of them to be able to shape and paint

How many tiles should we have at all?
However many your RAM can handle - you still have a cap, so working with Daggerfall sized worlds requires some wee-bit of switching between several tile sets. You can create sets/regions and pack them into folders within the Levels panel though, so it’s not too bad.

Can we display imposters in the place of levels not streamed in yet?
The link above actually covers exactly that - but you best take it from step 1 to make sure everything else is correct as well.

My earlier attempts resulted in a lot of sitting around staring at UE while things either loaded or messed up:
Pro tip - get used to it. Especially when you are messing with the landscape material. “Compiling shades” will pretty much your UE4 life for a few days.

Additional things you WILL have to worry about
World Origin Shifting.

I’d do it in C++ to get good performance, serve side / network replication etc.
But here’s a BP version for it:
https://forums.unrealengine.com/community/work-in-progress/1494046-eternal-apex-ambient-cyberpunk-megastructure-endless-rider

As to why you need to worry about that - as far as I understand it anyway - if you get too far from the center point of the map the physics calculations loose accuracy to the point they twitch.

6

@neighbourlee That’s what I thought, thanks for confirming it. It feels like the HLOD system should be able to generated streaming level imposters.

The next question then is how to streamline what we do next. I could create low-res static mesh imposters, or maybe we could load a lower detail landscape with the same tile size and display that turning off the relevant landscape component as the level that replaces it streams in (and turning it back on when the level is unloaded). That doesn’t solve the issue of foliage, buildings and other features placed in the level. I think we need some kind of level imposter.

[USER=“3140864”]MostHost LA[/USER] awesome, cheers! I’ll check that out now. Man, it looks like everything it needs is there, just hard to find.

I’m well familiar with origin shifting. It’s not just physics - even world position values in materials start being screwy.

7

[USER=“3140864”]MostHost LA[/USER] awesome, cheers! I’ll check that out now. Man, it looks like everything it needs is there, just hard to find.< Indeed, ty @MostHost hopefully this will get us away from game wreaking pop IN;)))

" My earlier attempts resulted in a lot of sitting around staring at UE while things either loaded or messed up:
Pro tip - get used to it. Especially when you are messing with the landscape material. “Compiling shades” will pretty much your UE4 life for a few days. " , Ya, thx for that, that alone feeling is dissipating…>…>…

P.S.__I knew about that TUT, but turned off by weird character,OOPS, should have continued for the later GEM o_0 <sigh> .

TY

8

That’s actually going pretty well, but there’s issues.

  1. You can LOD multiple tiles at once (one LOD at a time) but the tiles have to be loaded. For big worlds this is not ideal. I have 4000 tiles.
  2. There’s no process by which UE lets you set up the LODs for all of the selected, unloaded tiles, save it, then automatically load them one at a time and LOD them before unloading them. So my choice is to load 10 tiles at a time and repeat that 400 times, or use bigger tiles. Which brings us to…
  3. Use the biggest tiles possible.

And as the video points out, the LODing is pretty rubbish. UE seems to simply have one of the worst CLOD terrain implementations I’ve ever seen. So this is a stopgap measure at best and the whole landscape geometry system is not very good at its foundations. I suspect the reason is too many systems now rely on landscapes being a regular grid of vertices instead of something smarter.

I’ll keep going and figure a few more things out.

9

Well, it’s a landscape. It can’t be more complicated then a mesh.
however the 4.23 upgrade - when it works as it currently does not - introduced Landscape Layers, so you know. Things might already be Vastly different.

Why can’t you do 50 tiles at a time?
or all of them? Are you short on ram or what?

Using bigger tiles is not always a valid answer. Tile size some times depends on platform you design for.
you wouldn’t really put the biggest tile in a mobile game for instance. It takes way too much overhead to keep it on screen at all times.

Landscape tiles also have internal tiles btw, which LOD automatically based on distance as well - or occlude entirely when not in sight.

You can try to freeze the rendering and see what happens on your current landscape when you turn around for instance. Chanches are there won’t be much past the tile you are standing on.

Immidiate MS/FPS guzzlers are large meshes with no LODs. I learned this the hard way with the Ocean I added.
since it never culls it brings you down from a nice 60fps plus to a good 17fps or less.

Other than all of that, the while landscape system could use a overhaul when dealing with the layer paint, the fact it’s just 1 gigantic shader, the fact you cant have more then 13 samples at one time…

I was hoping 4.23 would address most of this with virtual texturing. Unfortunately it’s still pretty bugged so it’s not really worth the time to go and port the current project over.

For me, the biggest drawback of the landscape is the inability to have the player modify topology at run time.

same issue with meshes really. Why do I have to jump through hoops to port the landscape tools to the player when the tools already work and do exactly what you would need them to do?

Better yet, why cant I just randomize a matrix and create a random landscape during begin play? This was possible in any early game… and yes. You create impossible topology. That’s the point of it too…

10

I have 64GB of ram so I shouldn’t be running out. I do notice that it runs out of video memory while creating the LODs for the entire landscape though, even on an RTX 2080 Ti.

And yes, we know how the standard landscape component CLOD works. It’s just odd that they haven’t applied the same mechanism to the LOD generation or forced it to stitch the sides. The best solution apparently is to maintain the edge density and not reduce it at all.

11

TLDR summary on OP’s questions:

  1. Generating and exporting
    Whatever tool works. No issues here.

  2. Importing into UE
    Tiled World Composition import. No complications here too.

  3. Effectively texturing the landscape using generated maps like flow and normals
    As of current state of events(4.23) Efficient and decent texturing of large world landscape is impossible in stock engine.

  4. Managing it in the editor and in-game.
    In game management is simple. You want LOD setup with only 4 nearest landscapes rendered, while other tiles displayed as static mesh level LODs.
    When deciding tile size, favor as many tiles as you can get, while staying reasonable with total number of tiles in view.
    Typical considerations involve target vertex density and world size.

Id editor management with world composition proven to be royal pain. This part favors least sublevels you can get, contrary to in game.
That basically puts you between two fires on decision.

Overall, if you have resources to develop such world, consider rolling fully custom solution instead.

12

So there’s no way to use VT to get infinitely large textures? That’s a shame. It’d be nice at least to be able to define a collection of textures to use per component.

No resources for custom solutions sadly, I’m just one guy. I’m pretty much intending for a fairly minimalist world, it just has to be large.

And yeah, everything else worked great so far. My weight maps aren’t working for some reason though, and I can’t figure out how to scale the final landscape vertically without reimporting and going through the entire process again. Is there a way to do this?

13

Not that I’m aware without re-importing the heightfield. You can try to scale the Z of all the tiles, but that will create terracing.
Also, if you created the 400 static meshes… that’s kinda it? you would have to re-do all of that.

regarding the VT, I’m just waiting that 4.23 works right to experiment. the obvious benefit of VT is that you can almost never run into the 13 sample limit if you do things right. At the very least it should occur a lot less - and probably be vastly more performant…

14

" and I can’t figure out how to scale the final landscape vertically without reimporting and going through the entire process again. Is there a way to do this? "
To note: As many,IMHO ,troubles that Lumberyard engine has, some things its doing real well are 32bit terrain support and just what you mentoned, you CAN scale terrain on Z non destructively.

I have a project there but waiting to see where it all goes, as UE4 does most of what I need, tho Unity is catchng up in a way where it btw allows you to dynamically add new terrain, stitched ( and level ), ( like adjacent in WC in UE4) without being in any specific editor function. I could SO use botht right now,we sometimes change our minds as we design and need that flexibility !

15

Yes, you have to load one of the landscape levels directly to access the landscape’s transforms. Then, when you load back into the persistent level and start loading in the other landscape proxy sublevels they should all update to have the new transform.

It’s been a while since I’ve messed with this stuff, so I don’t remember if there’s one main landscape level or if you can edit any of them and have the changes propagate to all other tiles.

16

Neat! Thanks ConverseFox!

17

Pretty much all of my questions have been answered so far - thanks everyone! New set of questions now.

I’m now wondering about virtual texturing. I’ve enabled it and it just works on landscapes, which is nice. But the specific task I’m trying to do is:

  1. Use a single material for all tiles (easier to import and set up, no manual per-tile stuff)
  2. Import high res detail maps per tile into a virtual texture
  3. Have everything automatically assigned

I’ve worked out so far that if I switch to a UDIM naming scheme for my tiles they should import to a single virtual texture. I haven’t tried this yet but I’m going to today.

Do I just reference this virtual texture (plus offsets) in the material? What do the LandscapeCoord nodes when used in tiled worlds? I guess I can just use world coordinates instead though, which should do the trick.

I’ll report back with the results.

18

Good progress. You can import UDIMs as virtual textures. Still doing it myself, but here’s how it works:

First, export your tiled set from World Machine. Then use a script to rename everything (I’ll upload one later when I’ve polished it a bit).

From there, just import the FIRST tile into UE, suffixed 1001. It should find all of the others and load them in as well.

19

Well, things won’t import as virtual textures. I had some messed up numbering that start at 1010, and that imported as a (badly arranged) virtual texture. But when all of the numbers are correct and start at 1001 it just imports the one image. Kind of annoying.

I’ve poured through the small amount of code that handles this and it seems fine, so I don’t know what’s up.

Edit: I just figured it out. The base names before the UDIM ID all have to be the same.

20

You can’t trust the Landscape Coords node with a tiled world build, and honestly, it seems like it’s impossible to move the tiles around as a group, even with the tricks I found online. They always snap back to their old position.

I ended up figuring out the top/left and width/height of the landscape and storing them in a material collection parameter and a blueprint so they can align the landscape materials and be used for other things later.

I’m processing the landscape tiles into static mesh LODs again now (mostly to maintain an uninterrupted view distance as much as performance) and then I’m going to figure out how much stuff I can put down on the world, whether I need to set it up in streaming levels as well, and how well I can HLOD everything. Then I’ll get back into the main reason for doing this: developing a better procedural foliage system.

I also discovered another thing: make sure you set manual pagefiles and jack your virtual memory up to around 100gb. Even if you don’t end up using it, it’ll solve the main reason why you might crash when processing lots of landscape LODs at once.