Hello 
To explain the context, material instance can keep references of textures not used anymore, i’m not sure if that’s a bug or a wanted editor behavior to keep the override when we modify the materials, but my main concern with that, is that theses textures get loaded along with the materials / material instances when in game, memory that we could use elsewhere, i did implement a pre-cleanup step in our chunk generation to get rid of material instances dead references like so
[Image Removed]
But this is a time consuming step since i don’t know yet at that point exactly which assets the build will require i need to get through all of them, i could probably easily implement this kind of cleanup in the loading serialization of the material instance, but is that gonna affect directly the asset registry, or do the references would still show up when i will do my generation of dependencies ?
Also, more complicated topic, but the material editor allow a cleanup graph, but my attempts to create an automated cleanup of theses have hurt some walls so far, the cleanup not beeing available outside the engine and tied to the material editor, i tried to reproduce the cleanup steps, but i always ended up with materials that needed recompile (this test was on 5.5 tho, not sure the state of this in 5.7), what would you advise on that side, is it worth to look into this even or simply resaving my materials after the cleanup will modify their hash and nullify my caches ?
[Attachment Removed]
Hi Alex!
There’s a couple different issues with leftover references that we know of, one already addressed, one that we’re working on.
Case 1, Unused Default Textures: Let’s say you have 2 texture sample parameter nodes in a material, both behind the branches of a static switch.
One of them will be unused, but both require a default texture in them. Before, both of these textures would be loaded at runtime.
This has been fixed in 5.8 with CL 49947282, with r.Material.StripUnusedDefaultTextures toggling the behaviour. Now, both will still be cooked but only the actually used one will be hard referenced and loaded. Removing the unused ones from the cook references as well is something we’re working on.
Case 2, Textures overridden in a Material Instance, which has had static parameters changed: If you override a textures in a Material Instance and change a static parameter which hides that texture, it will still be in the members of that Material Instance, silently getting loaded at runtime.
This is still being worked on but the main way I can recommend fixing this is at cook serialization time - by getting the UniformExpressionOutputs of all MaterialResources of a Material Instance and finding which texture is unused in any of them and removing its reference.
If this is about another case which I’ve missed, please let me know.
As for the Graph Clean-up, that just removes unused expressions, which will always require a re-translation and re-compile. Any of these changes will modify the StateId property of a Material, which will lead to invalidation in both DDC and Cooked Data.
[Attachment Removed]
Hello
Thanks, the first one would be a big win already actually, i’m gonna look at that right away My case was mainly about your point 2 indeed
I though about moving to the serialization step, that would indeed have the same effect as what i’m doing at the moment, but that would keep the packaging side, one way to possibly manage it without having to load the asset before the actual cooking would be to store the dead references in the asset registry, so that a tag could give the references to void out, like at the serialization but saving time instead of cooking time
Side question, but is there any plan to eventually do the same with materials and overrides for static meshs component ? it’s probably more complicated since that would be the static mesh loading the base materials and the static mesh component loading the overrides, they would obvisouly still need to be packaged, but that could reduce pressure on memory where they is a static mesh used only with overrides for example
I can live with the materials not cleaning up it’s mostly to day to day build when there is a lot of modifications ongoing, don’t want to bother the dev materials to cleanup unused pins, but your point will void that issue for me
Anyway thanks
[Attachment Removed]
It makes sense to do it at editor save time, as you mention. In PreSave would probably be a good place. I think doing it there would remove any reference automatically, since the Import and Export table for the package are built at editor save time and this would clean them out before that.
There is a function for determining the visibility of material parameters: FMaterialEditorUtilities::GetVisibleMaterialParameters(). It could be used as is to filter out all Parameters which are in a static switch branch that will be compiled out and remove them.
As for static meshes, there is nothing right now that I am aware of to address that, although it’s good that you pointed it out.
[Attachment Removed]
Yeah if the clean is done at editor save time that would indeed work for the asset registry as well, i think that’s my best way of dealing with material instances without taking too much time in the chunk generation step
I was wondering as well, do the materials graphs stored eventually as editor data ? if that’s the case, could it be possible to include a step in the material serialization itself, not to remove the static switchs of course but to clean up any node that is not pinned to something, basically a way to keep the graph with the unused node, but without affecting the cook ? I’m guessing that could be complex, if the graph is the data used during the cooking as well, it may requires detecting such unused nodes at saving time and move them to an editor only data for example, but then back to the graph at loading time in editor
[Attachment Removed]
Hello i tried to integrate the commit in 5.7, but i’m getting some mixed signals from the memreport
[Image Removed]Here we can see the UI textures and the unstreamable render assets doubling, but in the texture report futher down
[Image Removed]i can observe indeed a nice reduction for in memory texture, i’m wondering mostly about the doubling in the UI texture size and the shift of memory between the streamable render assets to the unstreamable render assets
[Attachment Removed]
Hi there!
I’m glad to hear it’s helping. I’m not sure what is happening for the UI textures here. It is possible for that to be variation between tests - there’s not really any way for them to be affected negatively by this, especially since they’re unstreamable assets.
The rest do show a nice reduction in the expected ranges. You do have to be careful with measuring this though - the streaming system will use any new free space it has so that can mask any gains you will have had from this in measurements.
‘ListTextures’ might be able to give a more accurate answer, for a specific test scene.
[Attachment Removed]
Yeah i though that the overall memory might not reduce that much due to stuff getting use of the memory freed The list textures is the second output indeed i’m gonna do more test about it, but it seems weird that the two values changed that much, the two memreport were taken after entering a map without opening any UI, but i will do more to see if it’s a tendancy or something did go wrong (could be only the stat of the first picture not accurate to reality, or i messed up some stuff during importing the CL). But thanks, that’s indeed a nice thing to have, especially with our memory struggles
[Attachment Removed]
Hey!
Yeah, it seems weird for memreport and ListTextures to be so different. The best indicator here is the number of textures - this will have a difference if textures are not loaded because of references which have been stripped during cook, no matter what the streaming system does to fill up the new space.
Do please let me know if the second suggestion (to clean up unused parameters) also helps. While there’s no CL I can point to for that one yet, I believe it can be a low risk way of getting back some memory by removing unused parameters which might include Texture References.
[Attachment Removed]
Hey Did not had time to test more memreport to see if the UI textures were test variation yet, but as your second suggestion, the first picture in the thread is actually what you suggest, i’m getting all theses references leftovers in material instances and remove them during the build process I asked one of the dev, and he was saying that material instances do not get dirty when a material is modified, which can make sense, but then that would make useless a system to strip it in editor save to me, since some references will be lefts, thinking about playing with asset registry tags during either the material instance editor save time and parent material save time, but that will be for later, i think my best way to handle that right now would be to process the material instance cleanup during the dependency walkthrough (instead as a pre-task) to prevent going through assets we never gonna see in game)
Another way we could make that possible in editor, without requiering an extra step, would be to walk through the material instances linked to a material during the save, if we detect any reference that should be gone, we could dirty the material instance, but that may be applicable in our project but not as default stuff, some people may have thousands of material instances that would require loading / check on material save
[Attachment Removed]
Hey Alex!
Your colleague is correct, MIs do not get dirtied if a base Material gets modified. This means that if you remove a parameter from a Material, the texture override can still remain in child MIs.
In the meantime, I was able to get the change in that cleans up unused parameters for MIs during cook.
This contains the first CL:
https://github.com/EpicGames/UnrealEngine/commit/7fc455614baa55bc63eac900140949fe9f812b9e
and this contains a second CL that was required:
https://github.com/EpicGames/UnrealEngine/commit/1ed7c6e718a792009c3cf8fd6c65c46d6a65046d
However, this is not perfect. While it tackles the problem mentioned by you, it only works on MIs which themselves have static permutations, as it needs the output of the translator. Any MI which might have overridden a parameter that gets removed but that did not override any static switch will still have the texture reference.
This is a compromise which gives some savings for a relatively free cost at cook time, but it doesn’t get everything.
I think the main way to 100% fix this issue is something similar to what you mentioned - a quick utility that goes through all MIs of a certain parent and checks if any overrides are leftover using FMaterialEditorUtilities::GetVisibleMaterialParameters(). If there is, it dirties the MI, removes the override and resaves it.
This probably could be a pre-submit step, rather than a Pre-Save.
[Attachment Removed]
Hey
Thanks for the commits and infos, i think i will create that tool again, i was doing it previously during the ci steps but i think that should be something to use maybe pre-release time to clean all references and not take extra time for normal operations
[Attachment Removed]
Yeah, I think that’s the compromise which can give the best wins for this problem.
Do please let me know how it works if you get the chance to test it and if there’s anything else I can help with.
[Attachment Removed]