Better way to check for duplicate textures?

Development Programming & Scripting C++
, ,
1

My project has over 2000 textures imported over the course of development. I suspect there are many duplicated textures and want to remove them.

Right now, the only way I see is to access each texture’s pixels (following instructions at UE4 - Reading the pixels from a UTexture2D - Isara Tech.), then computing and storing a hash for it. Then for each subsequent texture, generate the pixel hash, store it, and check against the past stored hashes for duplicates.

Is there a better way? UE4 doesn’t seem to have an easier way to check if 2 textures are the same.

I am doing this in editor mode and open to using C++, python, or blueprints.

1 Like
2

Interesting thought. If you do build a tool that does that, I’d be interested in it. Currently working in a project that’s had many, many developers and artists over quite a long span of time, and it wouldn’t surprise me if there’s a lot of duplicated assets.

3

I’m not an expert, but I know that the python editor plugin, while out of date, was built specifically with tasks like these in mind, and one of the only examples that exist for the api covers something very similar.

4

I ended up implementing the method I described and it worked pretty well. Uses both C++ and python, though I imagine it can be adapted to pure C++ too

C++ Code

int32 UPythonHelper::GetTextureHash(UTexture2D* MyTexture2D) {

    TextureCompressionSettings OldCompressionSettings = MyTexture2D->CompressionSettings; TextureMipGenSettings OldMipGenSettings = MyTexture2D->MipGenSettings; bool OldSRGB = MyTexture2D->SRGB;

    MyTexture2D->CompressionSettings = TextureCompressionSettings::TC_VectorDisplacementmap;
    MyTexture2D->MipGenSettings = TextureMipGenSettings::TMGS_NoMipmaps;
    MyTexture2D->SRGB = false;
    MyTexture2D->UpdateResource();

    const FColor* FormatedImageData = static_cast<const FColor*>(MyTexture2D->PlatformData->Mips[0].BulkData.LockReadOnly());

    TArray<FString> pixelList;

    auto sizeX = MyTexture2D->GetSizeX();
    auto sizeY = MyTexture2D->GetSizeY();
    for (int32 X = 0; X < sizeX; X++)
    {
        for (int32 Y = 0; Y < sizeY; Y++)
        {
            FColor PixelColor = FormatedImageData[Y * sizeX + X];
            pixelList.Add(PixelColor.ToHex());
        }
    }

    MyTexture2D->PlatformData->Mips[0].BulkData.Unlock();

    MyTexture2D->CompressionSettings = OldCompressionSettings;
    MyTexture2D->MipGenSettings = OldMipGenSettings;
    MyTexture2D->SRGB = OldSRGB;
    MyTexture2D->UpdateResource();

    return GetTypeHash(FString::Join(pixelList, TEXT(";")));

}

Python Code

import unreal

lib = unreal.EditorAssetLibrary
editor = unreal.EditorLevelLibrary

asset_reg = unreal.AssetRegistryHelpers.get_asset_registry()
record = {}


helper_class = unreal.load_class(None, '/Script/Test.PythonHelper')
helper = unreal.new_object(helper_class, editor.get_editor_world())

filter = unreal.ARFilter(class_names=['Texture2D'], recursive_paths=True,
                         package_paths=['/Game/GameAssets/'])

assets = asset_reg.get_assets(filter)

with unreal.ScopedSlowTask(len(assets), "Processing Textures") as slow_task:
    for asset in assets:
        if slow_task.should_cancel():  # True if the user has pressed Cancel in the UI
            break

        slow_task.enter_progress_frame(1)

        name = asset.asset_name
        texture = asset.get_asset()
        hashNum = helper.get_texture_hash(texture)

        if hashNum not in record:
            record[hashNum] = asset

        else:
            print(f'Duplicate found. {asset.object_path} is the same as {record[hashNum].object_path}')
            lib.consolidate_assets(record[hashNum].get_asset(), [texture])
2 Likes
5

That looks pretty spiffy! Thanks for sharing!

6

2 functions
check same pixels on 2 textures - UTexture2D::HasSameSourceArt
get crc from texture - UTexture2D::GetSourceArtCRC