Weirdest texture compression bug I've seen. Need help!

[Kligan]

I’m working on viking weapon sets. Recently got to making shields. The base idea was to make couple bland wooden shields and then apply paint texture on top of it via masks so I could use material instances later to apply variations of shield patterns.
I began to work on the first pattern and used simple black and white mask to just apply paint over the wood to see how it will look and start making more complex texture masking later.
A-a-and… Strange thing happened. The colorful pattern texture was… weird.

To fully describe it, let me show you something.
So the pattern itself is a wyrm with golden outline on blue background.
Here’s how it looks in photoshop:

http://i.imgur.com/AvokIMc.png

Now, I will disable golden outline and import it in ue4:

http://i.imgur.com/xlUaFrB.png

Looks fine. Smooth lines.

This time, I will enable outlines layer in photoshop, and disable brown coloring:

http://i.imgur.com/CMDELay.png

That also looks fine.

And now let’s enable both outlines and coloring:

http://i.imgur.com/VvKjQe1.png

WHAT THE HELL HAPPENED?! (right click the image and open it in new tab to see what I mean)

This is how I first imported the texture in ue4 and saw weird pixels of blue and brown coming from under smooth golden outline.
When I begin to try to figure out what causes it, I discovered, that if brown coloring is separated from golden outline, it looks fine. But when they are together - it’s all messed up. Which doesn’t make any sense! Why is this happening?

I tried different image formats - it doesn’t matter. I tried different compression settings in ue4 - on some it fixes the issue, but they increase file size and don’t work with applying to mesh as texture.

If anyone has any idea about what is this ■■■■, please share with me! I’m stuck.

[Kligan]

Hey!
The material is very simple so far:

The problem is in texture itself once it gets imported in ue4, as you can see here (again, open in new tab to see):

For some reason, it somehow smears the brown and blue colors, while keeping golden outline intact.

[Kligan]

Increasing the resolution of the pattern texture to 2k fixes it… but obviously I want to avoid doing that for optimization reasons.
Nothing else works to fix it and I don’t know what to do

[Luos (Luos_83)]

ok, set anything thats just grayscale to “alpha” compression setting, its one of the more crisp settings. also for those textures with the “alpha compression” setting, only use the red output, no need for the RGB output.
that should help with the crispness without the need to go to 2k.

for additional compression tricks for masks etc look at: UE4 - Advanced Mask Texture Optimisation Tutorial - YouTube

In your case you might not want to merge the grayscale channels into an RGB setup but keep em seperate grayscale’s set to alpha compression.

If the files are rather small, you can decide to remove mipmaps as well, though shouldn’t be needed.

[Kligan]

The problem is not mask texture. The problem is the pattern RGB texture as can be seen on the previous screenshot. It’s not that it looks bad on the mesh, it just looks bad after compression, the texture itself. And only when both golden outline and brown coloring are there. If I disable either of them in photoshop, pixelisation of blue-brown disappears.

Edit: and thanks for the video, I will check it out.

[Luos (Luos_83)]

did you make sure that any of the mask textures dont have SRGB enabled? that will cause a lot of crosstalk between rgb layers.

sadly I dont have the time to do a hangout call atm, else i’d help you out that way. you can always join Discord (discord channel) and ask there if you feel like more help is needed.

[Kligan]

Yes, masks are all fine. They do not interfere with textures at all. As I’ve mentioned before, I narrowed down the issue to compression messing up the texture for some reason.
Thank you for the discord link.

[anonymous_user_c94f194d]

This is expected behavior from DXT1. Understanding BCn Texture Compression Formats – Nathan Reed’s coding blog

Color channels are not compressed separately and instead each 4x4 block has two endpoints. So blue pixels just bleed over thin outline.

[Kligan]

Yeah, I figured it had to do with compression. Now I only need to figure how to work-around this…

[Kligan]

(I couldn’t figured out how to mark this comment as an answer and ended up deleting it by accident lol Here’s it again)

Ok, your video was very helpful!
Using your method I cropped the colorful pattern texture and thus increased its quality w/o increasing file size.

Here’s how material looks like now:

And this is the end result:

That is exactly what I needed! Now I can finally move on to making it look like painted wood, rather than just blue surface with wood normals

Thank you a lot!

[anonymous_user_fbe2d247]

Try to use BC7 texture compression.

[Luos (Luos_83)]

Hey Kligan, how are you creating the masks in ue4? is each mask a seperate texture or are you combining them in the RGB channels?

Care to tell us and perhaps show us the material setup so I/we can hopefully advice you a bit better.