Quadmesh Billboard Shader not working with instanced foliage.


I’m trying to implement the technique from this video Fluffy stylized trees tutorial, using quadmesh-to-billboards shader in Unity - YouTube into UE. It’s working as expected with static meshes, but when I use it with foliage painting it seems to break the screen facing rotations. I tried implementing any fix I found on forums pertaining to billboards, but none of them are using this technique and I can’t figure out how to combine the fixes with the technique. If anyone has any insight that would be greatly appreciated.

Here is the WPO section of my graph:

The shader you put together does absolutely nothing the tutorial shows.
Mainly I think this is because of the use of the transform instead of a multiply.

First. HOW you made the model matters.
the UV could be flipped in the material (the multiply -1,1 node) OR, they could be flipped in the UV of the model.

IN unity you have a Y up coordinate system. in unreal we have left hand coordinate system (z up)

The whole shader starts off with the “assumption” that the UV plugged into WPO or World Displacement generates a camera facing version of each face.
This isn’t true in UE4.
Really Not sure how to get that part going.

To rotate the position of each face you need, at the very least:
The Pivot Point of each face (given by the world location of the vector field transformed texture coordinate nodes transformed to worldpsace)
A rotation axis. where the camera and Atan2 come in to figure out the rotation to apply.
if you have those 2 values, plugging them into a Rotate about Axis along with 0,0,1 as your normalized rotation will yeld a “billboard” like effect that follows the camera.

I’ll try and take a stab at this tomorrow because it’s an interesting effect. Wouldn’t hold my breath on it working as easily as it does in Unity though.

The shader does do what the tutorial shows. The author of the video even posted the Unreal version that I used - you can find it on this link Imgur: The magic of the Internet.

As I said, it works with static meshes, but not with painted foliage. I put in my mesh, created as shown in the tutorial, and applied the material and all works well. Painting that mesh with the foliage painter tool breaks the effect though. The painted version still moves based on camera perspective, but the rotation seems incorrect.

In the picture, you can see the bushes in the foreground look nice and fluffy. Those were manually placed. In the background, you can see many parts that are facing left and resulting in a very thin look with various bald spots. Those were painted with the foliage painter.

Its probably because of instancing. I’ll give it another go later on. Your code from first post didn’t do anything at all to the mesh displacement for me.

Converting the UV from Camera space to Local is probably the issue though. Because of instancing.
odd, Because world to local works for me on vector fields.

But hey…
The position x/y/z of the quads is mathematically
absolute world position + the (uv×2)-1
z can be just from world position.
To keep rotations convert the uv from local to world before adding them (or subtract) to absolute world position.

The pivot point is literally (uv×2)-1 local to world.

Give rotate about axis a shot with that…

Thank you. I’ll be trying to figure it out. I’m not much of a technical artist, but I’ll do my best.

Tsk. Technical artists aren’t supposed to know this kinda stuff either XD

This is much closer to programming, or being a mathematician with multiple PhDs, one being in quantum mechanics :stuck_out_tongue_winking_eye:

Sorai, do you get alot of billboard clipping

Thats my current result, Looks fine as a screenshot but when i rotate around the foliage ball you can see different billboards clipping through others so what you see is colors rapidly change like poping

Here you can see it clear the planes intersecting

This is my thought

  1. The dot represents the camera and the hex shape is representing faces of a mesh be it a sphere ect
  2. Shows the faces as billboards facing the camera
  3. Part of the effect makes the faces bigger this is what leads to clipping and intersection where all the billboards are at different angles
  4. This is what i thing would be better, have all the faces/billboards using one angle to face the camera and not per face. Something like the angle from the mesh as a whole to face the camera
  5. So even with there bigger size there less likely to clip and if they do like shown there would be faces in front covering it for the most part

The result would be all billboards are at the exact same angle wile retaining the central position as before, this i at the very least should reduce clipping/intersection if not remove it entirely

No idea how to do this thought but im hoping this can be resolved as i really want to use this method

Id also love a way to ‘override’ the mesh uvs to in the material so you dont actually have to edit the meshes uvs
Atm we have to edit a mesh to make each face fill the square uv bounds, i would love a way to do that in the material to cut out the manual mesh edit

Well, unwrap the mesh from a sphere before hand. So that one face fits the full UV using sphere unwrap/follow active quads.

Then each individual UV is just a UV - frac.
and you don’t need to worry about how the object unwrap or editing the UV past that.

I know that i mean id like a way to do this material side so that the mesh can have a normal uv texture map but then also be used with this foliage billboard
I have a specific use for this multi purpose use, I mean i can just make two uvs maps for the mesh one normal and one for billboards but im looking or hoping for an automated way to do this material side

Ideally anway

Having been looking into the camera facing thing i talked about im not too sure it will work the way i think not without seeing it

If instead there was a way to blend faces at there intersection to avoid the hard edges, like a sort of depth fade
Or someway to control which faces render over each other. Like when you get meshes to show through a wall mesh ect

For my example all my faces are of one single color in a gradient down, if i could get all faces of each color to render in order down the gradient
So lighter colors always overlap darker colors faces of the same color would still intersect but you wouldn’t notice as there the same color

So some way to blend or order what face visually overlaps which would be perfect
Could be something like moving faces closer or further from the camera to create the order
So the lightest faces move closer to the camera by x distance and the darkest moves away from the camera by x distance
Should create a visual order of which colors overlap which and prevent noticeable clipping of faces of different colors

I think to size up the faces, this is naturally going to happen. For me, it’s an effect that I want and as a static mesh it looks exactly how I want while rotating from all sides. I still can’t figure out the foliage painter version, so I might have to put that on the backburner.

If you want a clearer distinction in your gradient, you can probably experiment with the mask texture. The only reason you need to size up the faces is to create a “fuller” bush, tree, whatever. If your mask has less empty space in it, your foliage will look less sparse when not sized up, so you can prevent overlap and have a full foliage. Maybe even layering the mesh, meaning putting a shrunk version of the mesh within to help fill any empty space resulting from not sizing up faces. This second mesh can either have the same material or just a basic tiling material with no opacity that uses a similar pattern.

Re the clipping issue.

the material should probably not cast shadows at least on itself.

if you need shadows from it. Probably use 2 meshes. One that casts a shadow and is invisible one that doesn’t cast a shadow and is visible.

Another option that could help is to cast shadow as masked. Which will reduce the amount of shadows on the cards - if you use a masked material.

thats about it.
obviously 2 meshes for one tree is only possible if_when using actor foliage.

Any ideas why the original version of the shader doesn’t work a lick for me?
The UVs are all correct. The mesh quads are actually separated. They can be tilted /moved independently. Just not using the shader code from either link given.

I assume it’s something to do with the model…

Can you show me what you’ve put for your shader?

You can try my mesh/texture and see if it’s the mesh:!AjxGLPe4N-8TgX7SG-JkXXlzhpwh?e=ye8cMo

pretty much identical, but a different transform because the UVs are rotated differently.

Trying your model right now. thanks for the share.

Sure enough, it’s the model…

Have you made any discoveries by chance in regards to the foliage painting version?

Been swamped working 16 hour days lately, so haven’t even had the time to play with my old mesh to figure out why mod 2 works but frac doesn’t.

my guess is still that with foliage (instanced meshes across a wide area) the transform nodes don’t work as expected.

however, there’s 1 more possibility that I realized just now.

the instanced foliage could be rotated differently in every instance.
the values on camera to local wouldn’t be the same in this case. Throwing the rotation off.
So, it might be as simple as removing random rotation…

Oof, hopefully work eases up soon.

I did originally think the random rotation is throwing things off and I tried it before without, but it was still the same issue.

@Sorai try to use this:

And what would the inverting of the UV do in theory?

Inverting UVs is optional, this gives more disturbed billboard (atleast for our trees).

Main part which fixed for us the billboard effect for instanced meshes, is to change ‘CameraSpace to LocalSpace -> Normalize -> LocalSpace to WorldSpace’ to: ‘CameraSpace to WorldSpace’. This fixed instanced meshes and resulting rotations were finally correct with random yaw rotations.

I’m not a pro in materials part, just wanted to share the solution (which is working for us).