Camera facing foliage for trees via material

[Kelda]

Hello dear people,

for the last five days i desperately searched for a way to make my tree foliage align with the camera direction via the material.

Unfortunately the math-hall (2.21) solution doesn’t work because it shrinks all billboards and merges them onto one vector.
Epics stylized foliage material solution doesn’t work either, because it only rotates on one axis but more importantly, it rotates all billboards around the object center. But for the right effect, each billboard should rotate around their own pivot point.
Im afraid we can’t use a particle system either for several reasons, one being the algorithm we made to place the trees as instant static meshes.

This GIF of Test Drive Unlimited shows exactly the effect im trying to archive via the material.

But the closest i came so far is this effect, below (made by Pontus Karlsson on YouTube). His effect works perfectly fine but unfortunatly only with UV spheres, causing in a stylized look, that doesn’t fit the style of our project.

If anyone can help me out here, you couldn’t imagine how thankful i’d be!

Best regards, Jan!

[brisck1]

Bumping this old thread as I’m also searching for a solution to this!

[Quittzer]

did you ever find how to do this?

[sarahlenker]

@brisck1 @Quittzer Hmm… the thing that makes this so tough is the tex coord to world position translation, since each billboard location needs to be evaluated individually. While I’m not sure of the method for that, I might have an idea for a workaround.

Most billboard shaders do a comparison between the object position and the world position, and this is where something like leaves on a tree will fail, since the object position will be the tree base. Using instances for the leaves could solve this issue, since then you can use instance position instead.

OR… what if you store the leaf offset from object position in the vertex color of the 4 vertices of the leaf billboard? This will localize the rotation values and should give you the desired result- admittedly with a lot of setup. I believe vertex colors clamp the floats though, so you’d have to set the location/1000 or some similar number.

What that logic might look like:

Replacing the object position in a shader such as this:

How to make Camera Facing Billboards (UE4, valid for UE5)

And in your DCC, a leaf billboard with a pivot location of -300,500,150 would have its vertices saved as vertex color -.3,.5,.15. If you have the coding skills, a process like this should be very easy to automate through Python or Houdini.

[Quittzer]

@sarahlenker not sure i understand, but here is my node setup ive already got, it works great for single planes like the sun in my scene but not for trees

[sarahlenker]

Yeah, so those Object Position nodes are where you would replace it with the logic I sent. The solution I sent though will require you to make changes to your trees in a 3D modeling program.

Here is an example tutorial on how to set up vertex colors in Blender

BLENDER - VERTEX COLORS TUTORIAL

As I mentioned earlier, you would want to set the vertex color value equal to the billboard pivot location (assuming the tree’s pivot location is set to 0,0,0).

The reason your current billboard setup doesn’t work with trees is because the Object Location is grabbing the pivot point of the tree. In reality, to move the leaves, you need to grab the pivot location of the individual leaf planes. This solution allows you to do that.

Let me know if there’s anything in particular you need explained more in-depth.

[Quittzer]

thanks! i appreciate you explaining this, i do got a couple more questions. do i need to paint every leaf card a seperate color? or do i make all the leaf cards one color? here is my tree model, maybe you can point me more in the right direction on how to do this.

[sarahlenker]

Hi there! Sorry for the late reply, this missed my notifications.

Sorry for the poor quality haha, I don’t have my Wacom tablet right now. I’m going to try and break this down further so hopefully you can understand exactly why we’re doing this.

A is the world origin, and will be the origin of the tree when you import this into Unreal. If we create a billboard material and grab the Object position, this is the point that it will grab by default. That is why the billboarding fails, because it’s trying to orient itself around this point, which is relative to the entire mesh but not the individual leaves.

B is the origin of the leaf plane. THIS is the point that we want to orient the billboard around. So how do we make that conversion? Well, that’s where the vertex color comes into play. We can’t grab this world location because it’s pretty arbitrary, so we need to set it manually.

(You can skip this section if it’s going to confuse you, but I’m going to go into the full detail of how this works for your reference) So let’s say we set the vertices of a single leaf plane to the value of their pivot. Why do we need the logic in the material of Object Position + Vertex Color? Well, that pivot point is relative to the origin of the tree. If we moved a tree into Unreal and set its world location to 0,0,0, just using the vertex color would work. However, when you move the tree to a new location, this vertex color value will no longer be relevant. By adding it to the object position, our values stay localized to the tree and thus will always be correct.

So, what values are we setting and why? C in this diagram are the vertices of the leaf plane. Since we want the billboard to happen per-leaf plane, these vertices need to all have the same value, and they need to be the value of the plane’s pivot point (B).

So, how do we do that? I’m not confident that the vertex color can save large values, so we will need to make the value under 1. How do you do that? You divide your original number. In my example above, we’ll divide by 1000, then later in the material, multiply that back in to get the correct pivot location again. So, let’s say the pivot point B’s world location is 450, 233, 600. You need to paint the RGB values as .45 R, .233 G, .6 B. We do this because for all intents and purposes, a color is just a vector, and we use vectors to determine world locations, making the vertex color value compatible with the pivot location.

Let me know if that breakdown helped!

[kaidawai]

Also nota bene if you have an additional point you could also simply project.

[ x ] dot [n] =d

where n is the camera vector and d the distance to 0.
equivalently:

( [ x ] - [p] ) dot [n] = 0

n is again the camera vector whereas x and p are points.

[kaidawai]

Well I do understand your approach and like that idea to add information via vertex colors.
I still think we are missing something here.

Thing is: when we do not normalize and not remove the vertex position it does almost what it should. It does orient the face more in direction of the camera. I emphasize it does this to THE FACE and not the whole object. Wrong amount.

Yes, I was also thinking about scaling. And indeed with that you can achieve a real billboarding effect. And again: done to the individual face which do stay in the place. But that scaling unfortunately does not only affect the coordinate transformation but “everything”(whatever that is depending - It becomes to large, to small …. ). And that puzzles me. I feel like there needs to be a translation to I_DONT_KNOW_ATM and a scaling done there and then translation undone. Something the like.

Also this works somehow in unity(the video the OP posted) per Face(!) . And there are a lot of artist who got it working for special cases by adjusting thrown in parameters via trial and error in various places.

I think: The World offset of the vertex should not be removed because that is the only thing that gives the vertex position(face/uv island). But instead the billboard needs be applied to it. But before that the billboard needs to be translated, scaled (and probably again translated back).

Unfortionately: I can’t wrap my head around it. I get close to it but I always get something wrong.
Maybe I am wrong. But why would it work in unity and not in unreal : should be difference in the coordinate system, difference in World Position offset which should be possible to adjust.

[kaidawai]

Also while I do like what this guy does to explain how things work(great job there). https://www.youtube.com/watch?v=N9IO2YJBReo
What he does in this case feels awkward. First we undergo effort to get a transformation of the stacked uv coordinates(one coordinate has a multitude of mappings in the model). Just to throw everything away in the next steps and and use a linear interpolation of coordinates of a bounding rectangle of the object instead. That does not really make sense. For that we don’t need stacked UV in the first place. Actually it is kind of bad as the same thing is drawn again and again - multiple times with no visual effect: why?
Actually its only a coincidence that his(Visual Tech Art) approach seems to work. The coincidence is that the uv mapping has the shape of a quad and the bounding rectangle of the object is - well - a rectangle. Hence the uv mapping shape and the bounding rectangle match with little lep mapped scaling. Pure coincidence that we crafted the the uv isles as quads to simplify the uv mapping.

(Also while it is a great and working idea: abusing vertex colors for information we already have where it is supposed to be - in the uv mapping of the model - feels like there has to be a better way)

Also I mentioned https://www.youtube.com/watch?v=iASMFba7GeI works in unity and he also does it nicely step by step and shows what is suposed to happen(so you have that link too).

Its not like Pontus Karlsson invented something never done before. Still kudos:He reinvented or adapted. There are whole chapters in books about it. I do have GLSL code for procedural created trees that works. I have GLSL code that even maps a multitude of different textures in a uv atlas to different faces and billboards them. Its no real magic to apply a transformation matrix from view to world to some coordinates.

All that makes me pretty sure that it has to work. Actually my problem is that I do not understand: why doesn’t it in Unreal? It works kind of everywhere except for Unreal(Nothing wrong with Unreal: fantastic engine).
What makes the difference? What do I overlook? Banging my head on the wall

Doh: its sooooo simple. Unity has units of 1 meter. Unreal has units of 1 Centimeter. So the 1m stretching is a lot more than 1cm stretching. Also the size of the faces and layout does matter.

We do not do a full billboard but stretching the faces parallel to the camera and by that orient them more to the camera. A face of one unit length perpendicular to the view(on a cube left and right side) get rotated only by 45 degrees(assuming you subtracted -0.5 from the texcoord ).That is fine if the objects faces laid out nicely. There are some leaves in the middle of the tree with 45 degrees - so what? The ones in the front are nicely facing the camera as they should.

So simply take the different units in unity and unreal (factor 100!) into account already when creating the model and later also when scaling the texture. Also don’t normalize the texture coordinates: that kind of breaks everything. Remove that. Normalizing will destroy the anticipated square shape: you do not want that. You can do some normalizing(again take the unit sizes into account) as a final step though if it looks better.

With that it should work just fine. No mapping to the object. No normalizing of texture coordinates(breaks everything).

[formatting of this post keeps breaking when I send it.. but ok ]

[kaidawai]

What you could do:

• You can collapse the 4 points of your faces in the model to the same position(still having all 4 uv corners mapped to them). That gives kind of an odd model without the shader. But the shader will put the vertexes in place. Now that gives you 100% individually bilboarded square faces.

and then again you can play with that:

• You can give your uv any shape you like (any number of points - just make a face of a couple of points that all have the same position and in the uv make them form the shape you like) .
• Or have different islands of different shapes. Don’t need to be squares. Just collapse the points and stack all the islands above each other.
• if you don’t stack them you need to translate the texture offset per island which may turn out to be difficult.
• if you have different shapes you can also put them into different channels in order to map different textures to them.

and so on ….

Just do NOT normailze position vectors (get rid of all normalisation before the View/Camera→World transformation).
Rule of thumb: Never normalize position vectors. Only normalize direction vectors.

The shader

Its as simple as below. I added some optional parameters.

• Texturescale scales all the face individually by the same given factor(default cm→m scaling).
• zscale moves the whole object(all affected faces the same amount) away from or towards the camera.
• scale scales the whole object.

For the ops original problem:

sure enough everything stands and falls with the model. Where you place your faces does matter of course. Also selecting which faces should be billboarded and which not does matter. For partial billboard, the relation between the size in the model and the size of the square gives how much it is billboarded(zero size gives 100%). We do billboarding by scaling so faces do get bigger: take this into account for sizes of faces in the model(and if you do it the Pontius Karlsson way, the whole model: faces gets scaled so they need to be smaller in the model).

additional hint:

You could also use an uv atlas(have multiple textures for different leaves packed in one tiled texture) you can make inputs from parameters. Which texture you select is what you like choose:

I think that gives kind of a world of possibilities to create nice billboarded objects of all kind and using vertex colors proposed in a previous comment may add additional of room for creativity…