Perhaps. SpeedTree does export the atlas texture, it’s just that there is no billboard when I import it into UE4. I’ll poke around the compiler and see what happens. Once I’m done I’ll post updates for posterity’s sake. Thanks for all the help folks!
Edit: Update on the billboard stuff. I’m an idiot. Need to check “Both” when importing, not just “3D Models”. Now frond texture is black, but it’s a new issue! Hooray!
Here we go. FPS is definitely up, billboards are in place. My FPS is now anywhere between 30 and 40 depending on what I’m looking at (Probably depending on the amount of leaves in view). There are however still a few issues. I am going to try and copy / paste the billboard shader from the kite demo because the default speedtree billboard shader looks… well… bad. Another thing I can’t seem to resolve is this:
The mesh is smoothed in Maya, and I also tried setting the vertex normals to “up” manually, the fold is unfortunately still visible. Any advice is welcome. 
Edit: I am getting up to 50fps at times. Woo!
The kite demo one may be a bit complicated to take apart since its using pixel depth offset and sprite vertex shader etc…
The main issue with the default speedtree setup is that the normals are rendered from the tangent space of the “slice” of the mesh they are facing and since you usually see two slices for a given view there is always a harsh lighting seam down the middle. You can fix that just by disabling the Tangent Space normal option and then doing a transform of View->World on the normalmap texture. There are also more advanced ways to do the transform depending on the behavior you are after. Try that and see if it fixes the main problem.
The only issue with the view transform version is that it will be based on the absolute view angle, not the angle from the camera to any point in the world. For that you need to make a custom 3x3 transform using camera vector and the material function “create third orthogonal vector”.
One other thing the kite demo billboard material does is use the UV’s to create a cylindrical normal which gets overlaid on top of the texture normals which adds a bit of full 3d shape to the billboard that wasn’t picked up in the billboard render. Even if its not 100% accurate to the mesh, exaggerating the overall low-frequency shape of the tree in the normals can help the billboards read better as long as the difference is not too great from the base mesh. Sometimes I will actually do this by hand in photoshop by duplicating the normals and blurring the image more and more to feather out the shapes a bit.
No idea whats going on with the vertex normals, sorry. I am a max person so I am pretty unfamiliar with hard/soft edges in maya.
Hey Ryan,
Thanks for the quick answer. I have since done some more work. I was intent on just copying over the kite demo sprite material and use that, which I did, and got pretty decent results with even though I didn’t have proper specular or pixel depth maps (Use Normal Blue map as pixel depth). The issue with SpeedTree I had was the lack of shadowing so they’d stand out very harshly from the previous LOD. In an attempt to get the Kite Demo material over as faithfully as possible I tried importing another LOD4 to my tree’s static mesh, which kept crashing the editor unfortunately, with no minidump. I gave up on that eventually and just put the Kite Demo material onto the SpeedTree plane arrangement. With some tweaks to the round tangents, opacity offset etc. I got 'em blending in just fine.
I will however definitely check ou your SpeedTree solution to see if it’ll produce adequate results.
Thanks a ton for all your help, I believe this will be enough to get me started. I must admit I still have the nagging feeling that something’s wrong because my PC is quite powerful (780GT with an i7-4770) and getting over 60 fps on that scene doesn’t strike me as something that should be difficult. However, I’ll take what I can get and optimize more once we get our PS4 devkits.
Interesting. I wonder, does the shadowing get better if you remove the “speedtree” node from worldposition offset? That will prevent the on-edge tree slices from showing but it may be that is causing too much of the tree to be invisible for the shadowing pass?
We also dealt with a similar issue with our custom billboards. An engine change was actually made so that “camera position” was evaluated as “light position” during the shadow pass so the billboards would orient towards the light. otherwise the light would see an invisibly thin sliver of a billboard facing the player and thus cast an almost invisibly thin shadow if the light was at a 90 degree angle to the player-tree vector. Shadows so thin you cant even see them!!
are you testing this in a packaged game or editor? if editor, check your slate with stat slate
the editor takes a tremendous amount of fps and when you have new windows open apart from your map your framerate gets very bad
i was struggling optimizing a speedtree forest with kite demo grass to test, with quite alot of trees and density - i was getting around 70-80 fps in the editor and 120 in the packaged game because the editor was taking around 7ms
granted this was on a 970, but still - hope this can help maybe
Hey Ryan,
I have since gotten to the office I tried adapting the SpeedTree billboard in the way you suggested. I actually already unchecked tangent space normals but I didn’t do the View to World transform. Having done that, I actually like the SpeedTree billboard more than the Kite Demo one, it works better with the assets I got. However, there is a facing angle issue, seen here: http://imgur.com/a/PZ9np
It’s probably due to me not using the 3x3 transform you suggested properly:
I did figure out the rounded normals in the Kite Demo one and liked it a lot, so I added it to my “just the normal map” setup.
Looks like you have that a bit backwards. You don’t hook your normal map into the ‘create third orthogonal vector’ function. That node is used to create the basis for the transform to work. Only simple vectors go into that node.
Vector2 is usually just 0,1,0 or 0,0,1.
Also whenever setting up a normal transform, use a sphere normal map from engine so you can debug it more easily. Constantly place it next to a real sphere mesh and compare in ‘world normal’ view and take the screenshot into photoshop and view each R, G, B channel individually to see whats flipped etc. It always takes a few tries for everybody with this stuff.
I’ll post back some example images shortly.
Again, sound advice. These past 24 hours have been an avalanche of foliage related stuff for me and I am loving it. Looking forward to the examples! You can’t imagine how much of a help all this is, thanks a ton!
This is what the transform should look like:
Also make sure “tangent space normal” is disabled in the material settings.
“CameraPos” minus “ObjectPos” is used otherwise you will see the normals radiating around the camera when viewed from directly overhead.
FWIW I’m not surprised that the kitedemo billboard gave you odd results. Using pixel depth offset without a proper heightmap setup will make things worse than just disabling the PDO altogether. And also that material had a bunch of offsets tweaked just for that one tree that might make it more work than its worth to repurpose.
And also one more note about distant trees. Are you not using distance field shadows? I forgot to mention that our billboard shadowing problem only happened in a somewhat narrow range close to the camera since the trees were turning to billboards before the distance field shadow radius. But if you are using distance fields, out past your dynamic shadow radius the distance fields should be doing the shadow casting so the speedtree billboards should cast just as nice shadows as the real mesh. Pixel depth offset simply helps the billboards receive those shadows more accurately since otherwise they will be over-shadowed by the more accurate distance field shadows (since the billboard is always going through the middle and surrounded by the DF shell).
You may need to enable the ‘render distance fields’ project setting and also edit your foliage properties to allow the trees to generate distance fields. It will cost something but if you are able to significantly cut back your dynamic shadow radius as a result it may end up helping you optimize more.
And this is how I debugged the normals, by using a sprite with a sphere normal and comparing to an actual sphere in the world. The -1 on Y and which channel gets hooked to what in the transform was figured out by trial and error:
It does get a bit easier with practice since after while intuition helps you hook it up ~90% correct the first time, then just play with swapping things around until its right.
The selected mesh is the real sphere, the other two are the sprites.
Hey Ryan,
I do use distance fields and it seems that I accidentally did have my billboard radius further away than my cascading shadow distance so shadow casting was fine. I understand the PDO is for receiving shadows but the results I am seeing seem good enough. I spent some time today getting the actual LOD0-2 shadowing right.
I will most definitely run all my stuff through the sphere nprmal method to see if it’s all right first thing in the office.
Thanks again!
Edit: The second post attachment seems broken. 
Edit2: I figured it out though
&d=1436321691