I think I can help provide some feedback since I spent a good deal of time working on our Flora assets for our project, but really its a constant process of learning, testing and tweaking as we find something we don’t like or didn’t know and we try to improve on creating assets or in engine changes. I use a variety of source assets, and I also do use SpeedTree to aid in the creation of variants for each plant as well as to automate some things you will want for foliage. It is important to understand ST will aid in creation of the assets but it still takes careful creation of the trees, the quality of the textures, and the setup of the materials used, and also as importantly the scene lighting are just as important, so you really don’t need ST unless you want to simplify your workflow, what ST does can all be done with any modelling software and some practice and patience.
First for making more realistic trees there are improvements you can make in all the different areas. Polygons/Structure can be more interesting while the trunk isn’t bad the branches are very straight, but the worst culprits of low polygons is the leaf planes, there are not nearly enough of them and they are too large and flat for a good effect. You will want to find a good balance of poly’s for effect. Use your edges/loops on the trunks when you want to give it a turn and variation, but most of your poly’s are going to get spent on the leaf planes, typically 75% or more is leaf planes on trees.
Next would be Materials, you really should be using a separate set of materials for different parts, most important trunk, and branch/leaf sections, for a few reasons. Generally the trunk is a much simpler material similar to what you have shown in your images, but the Leaf material should be much more complex and is one of the places where we actually drive values into Specular input, it also use the Sub-Surf Shader model and we found we also have to shift some values based on distance from the trees due to our lighting settings. Here is our Leaf master material currently.
We have different masters for different parts, trunk, branch, leaf, billboard and all our trees have similar maps, and we tweak various parameters as we need, or make lighting updates.
Lighting for the scene will have a big effect on your materials and the look of things, we are building a dynamic world so we don’t use Lightmass and Light importance volumes as our lighting is dynamic, using DFAO, and a bunch of tweaked parameters, its an ongoing process that is a topic unto itself. It is not something we are totally happy with yet, mostly in the shadows and billboards at a distance. Some of our variable parameters are put there just to tweak things to play nice with our lighting. Its not really something I’m sure I can help give pointers on as its pretty specific to each projects needs, but if you are just doing a solo scene then our approach is not one you will want anyways as we are doing a big world with dynamic lighting, you will likely prefer to use Lightmass and turn on a lot of settings that can better work with that. But definitely look closely at your lighting settings when it comes to foliage as they will play nicely or not in different ways.
What are your current poly counts on your trees? We tend to run around 20-30k for average trees up close quickly going to 10-5k as you move away and billboard for surrounding trees. To get good looking trees your will need poly’s to create that nice look, and you will want to generate several levels of LOD, and Billboards to fill in the distance. In the example scene you can see where we have several LOD’s and Billboards actually in the partially occluded distant trees. You can’t really avoid having poly’s on good looking trees but you’ll be surprised how well a modern set of hardware handles polygon counts, another important detail is not to place any foliage manually into a level, they should all be Foliage Instances as this is much more efficient rendering of foliage types.
I attached a few images of a single tree showing the highest LOD0, and Lowest LOD2 before it goes to billboard. Notice it maintains shape pretty well but the leaf planes and trunk simplify a lot which is noticeable up close and effects the lighting a bit but at a distance doesn’t matter. Ideally though you want to go to billboard as soon as possible on trees in the distance to drop the load when they don’t need any details at all.
We do make use of SpeedTree nodes which are mostly driven by vertex colors which are baked into the tree part by the software, and you will see how in the master we have a few parameters for defining what part of the tree part the material is applying to, this is for the wind movement to behave nicely as leaves and branches will sway nicely but the trunk parts will only move a little for nice effect, this is something you can do manually in a modeller but its time consuming, same for making LOD’s and Billboards, which is the reason I’ve converted all our foliage to use ST because when you need to fill a world with foliage and iterate over them and make 6 variants of everything having the tool create all the combined map atlases, billboards, vertex color and LOD’s and packaging them into a single ST file for import and then working automatically with those ST nodes in the material is worth price of the subscription in time saved for me.
If I were to give any other suggestions I would give the ground a bit more definition and have the rock piles spread out a bit more, perhaps making the landscape seem to undulate a bit where the rocks are and having little depressions or runoff places to give it a bit more visual clutter. What we generally have found is if you provide enough little change here and there then the eye doesn’t get drawn to any one particular place or thing and it just feel more natural to the mind even if its the same thing as long as it doesn’t seem too repetitious you can reuse things and it will blend and work nicely together.
I’ve also included an image of a little glade near the village in our starting area, it needs more little details I think but its a decent start and shows the foliage we are using fairly well.
Hope that helps! Good luck with your scene, looking forward to seeing next iterations of your trees!