No matter what fancy videos come out showing off new game engines, nothing will beat tried and true techniques to get your game simulating a large world.
Firstly, never trust tech demos. The more artsy and cinematic the tech demo, the more your suspicions should be aroused that whatever is being shown would not realistically be in a game, which requires multiple actors, physics, particles, and all types of collision, if you want some action happening. Shadow of the Colossus on the PS2 was allowed to look artsy and cinematic and push the visuals of what’s possible on the PS2 because they designed the gameplay to not require a lot happening at once - just you, your horse, and 1 other colossus. And maybe a lizard or two.
If you want to push a game to the limits of what we’re seeing these days (eg. a giant lush forest) you’re going to have to rely on tried and true techniques - a one-fits-all game engine solution will never solve it for you, since you will have to make decisions on what’s important to your particular game, and cut out the rest, to stay on the cutting edge of what’s possible.
When I see screenshots like this, with some in-engine feature cranked up off to the horizon, I know you’re bound to run into trouble. There is no way for the designer of that feature to know what you need. Are you engaging in hand to hand combat with enemies only a few meters away from you, or is it a siege simulation where you’re lobbing projectiles off into the mountains in the distance? The collision for these situations will have to be handled differently, and trade-offs will have to be made to keep the game running fast.
In your case, you need to come up with your own way to break up the landscape so the game is only thinking about what’s immediately possible to interact with the player at any one time. Can you climb trees in your game? If not, then a simple cylindrical collision for each tree trunk will do. And only for the trees immediately around you. How do you only check the trees around you, and not every tree in the land? This is where you need to put in your effort to make your game run fast. If you are in a small square of terrain and it is the “active” square, then you only have to check the trees within that square. What about when you get to the edge of the square? If there’s a tree in the adjacent square right in front of you, you could chop through it until you cross the border. More importantly, any enemies would be able to walk through the tree. So you’d want an area of 9 squares where you’re in the center square, and each surrounding square is also included in the active set. If you cross out of that square into another, then that one becomes the center square. Then you can have enemies enter the area and move around as long as they’re in the active squares. If they leave the active area, you then have to decide how important it is to your game to keep simulating them. If it is, you will need to code up an alternative, simple simulation that keeps the enemy moving until he catches up to you and re-enters the active area. You could probably forego tree collision in this simplified simulation, and just rely on the heightmap of the landscape. Although if in your game you put up a barrier that should stop the enemy from moving through, this would have to be factored in.
So as you can see, it all depends on what you need to happen in your game, that determines how much work you have to do to keep it running fast and giving the player the sense of being in a larger active world. You can also see that most of your time will be spent on dealing with what happens in these border cases, when ever the active area moves (because the player moved into a new square) or a character enters or leaves the active area.
Every game company that made an open-world game (Elder Scrolls series, GTA series, etc.) has to deal with these issues, and your game is no different.