Additional Asset Directories not Copied to Packaged Build

hyperdr1ve · June 12, 2015, 10:50pm

My Info:

Windows 8.1 64-bit
Visual Studio 2013 Update 4
Unreal Engine 4.8 (binary from launcher)
Project was created in 4.7 and updated to 4.8
[DefaultGame.ini][1]

I am not able to get the Additional Asset Directories to Cook feature working in my project. Here is how I am validating this result.

Add an asset directory that I want to cook.

This asset directory contains textures that I reference in C++:

They are initialized in my constructor via a static like so (all works fine in PIE):

struct FConstructorStatics
{
	ConstructorHelpers::FObjectFinderOptional<UTexture2D> T_Face_Button_Left;
	ConstructorHelpers::FObjectFinderOptional<UTexture2D> T_Face_Button_Right;
	ConstructorHelpers::FObjectFinderOptional<UTexture2D> T_Face_Button_Top;
	ConstructorHelpers::FObjectFinderOptional<UTexture2D> T_Face_Button_Bottom;
	ConstructorHelpers::FObjectFinderOptional<UTexture2D> T_Left_Shoulder;
	ConstructorHelpers::FObjectFinderOptional<UTexture2D> T_Left_Trigger;
	ConstructorHelpers::FObjectFinderOptional<UTexture2D> T_Right_Shoulder;
	ConstructorHelpers::FObjectFinderOptional<UTexture2D> T_Right_Trigger;
	ConstructorHelpers::FObjectFinderOptional<UTexture2D> T_DPAD;
	ConstructorHelpers::FObjectFinderOptional<UTexture2D> T_Left_Analog;
	ConstructorHelpers::FObjectFinderOptional<UTexture2D> T_Right_Analog;
	ConstructorHelpers::FObjectFinderOptional<UTexture2D> T_Start;
	ConstructorHelpers::FObjectFinderOptional<UTexture2D> T_Special;

#if IsPS4
	FConstructorStatics()
		: T_Face_Button_Left(TEXT("/Game/UI/GamepadIcons/PS4/PS4_Square")),
		T_Face_Button_Top(TEXT("/Game/UI/GamepadIcons/PS4/PS4_Triangle")),
		T_Face_Button_Right(TEXT("/Game/UI/GamepadIcons/PS4/PS4_Circle")),
		T_Face_Button_Bottom(TEXT("/Game/UI/GamepadIcons/PS4/PS4_Cross")),
		T_Left_Shoulder(TEXT("/Game/UI/GamepadIcons/PS4/PS4_L1")),
		T_Left_Trigger(TEXT("/Game/UI/GamepadIcons/PS4/PS4_L2")),
		T_Right_Shoulder(TEXT("/Game/UI/GamepadIcons/PS4/PS4_R1")),
		T_Right_Trigger(TEXT("/Game/UI/GamepadIcons/PS4/PS4_R2")),
		T_DPAD(TEXT("/Game/UI/GamepadIcons/PS4/PS4_Dpad")),
		T_Left_Analog(TEXT("/Game/UI/GamepadIcons/PS4/PS4_Left_Stick")),
		T_Right_Analog(TEXT("/Game/UI/GamepadIcons/PS4/PS4_Right_Stick")),
		T_Start(TEXT("/Game/UI/GamepadIcons/PS4/PS4_Options")),
		T_Special(TEXT("/Game/UI/GamepadIcons/PS4/PS4_Share"))
	{
	}
#elif IsXboxOne
	FConstructorStatics()
		: T_Face_Button_Left(TEXT("/Game/UI/GamepadIcons/XboxOne/XboxOne_X")),
		T_Face_Button_Top(TEXT("/Game/UI/GamepadIcons/XboxOne/XboxOne_Y")),
		T_Face_Button_Right(TEXT("/Game/UI/GamepadIcons/XboxOne/XboxOne_B")),
		T_Face_Button_Bottom(TEXT("/Game/UI/GamepadIcons/XboxOne/XboxOne_A")),
		T_Left_Shoulder(TEXT("/Game/UI/GamepadIcons/XboxOne/XboxOne_LB")),
		T_Left_Trigger(TEXT("/Game/UI/GamepadIcons/XboxOne/XboxOne_LT")),
		T_Right_Shoulder(TEXT("/Game/UI/GamepadIcons/XboxOne/XboxOne_RB")),
		T_Right_Trigger(TEXT("/Game/UI/GamepadIcons/XboxOne/XboxOne_RT")),
		T_DPAD(TEXT("/Game/UI/GamepadIcons/XboxOne/XboxOne_Dpad")),
		T_Left_Analog(TEXT("/Game/UI/GamepadIcons/XboxOne/XboxOne_Left_Stick")),
		T_Right_Analog(TEXT("/Game/UI/GamepadIcons/XboxOne/XboxOne_Right_Stick")),
		T_Start(TEXT("/Game/UI/GamepadIcons/XboxOne/XboxOne_Menu")),
		T_Special(TEXT("/Game/UI/GamepadIcons/XboxOne/XboxOne_Windows"))
	{
	}
#endif

};

After packaging the build, I expect this folder to exist in the final package folder, it does not.

Am I misinterpreting the asset directory parameter? How do I get these additional assets into my packaged build?

Things I’ve tried:

Creating a User Widget that uses these textures as images. I then create a reference to that User Widget in my main HUD but again there is no change in packaging.
Copying the .uasset files manually to the packaged folder, this results in a [crash in GarbageCollection.cpp][5]. I’m unsure if the assets are problematic or something else.
Creating a .uasset file in the GamepadIcons folder, thinking that perhaps an empty folder was preventing the inner XboxOne folder from being scanned.
Using the Cook Everything in the project content folder option
Every variant of 32/64-bit development/shipping/DebugGame
Packaging in editor
Packaging via UnrealFrontend

Here is the [log output from packaging in editor][6]
None of these attempts resulted in any change in output. Please tell me what I’m doing wrong here, I’m sure it’s something fundamental or alternatively something is fundamentally broken.

hyperdr1ve · June 20, 2015, 2:51am

I’ve added an additional firehose worth of extra details. This issue is killing my studio.

hyperdr1ve · June 20, 2015, 5:51am

I don’t think these are related. My issue has to do with content not being copied.

Renderdog · June 20, 2015, 4:28am

I am not sure this is related. I just tried 4.8 today. I cannot compile a default scene in 4.8 without fatal errors. When I do a build it is blowing away a folder it “appears” to be looking for. Here is the link to my thread.

hyperdr1ve · June 20, 2015, 6:17am

Another note. After packaging completes, if I look in Saved\Cooked\WindowsNoEditor, I see the missing content there. So it seems that the packaging step is where things are failing.

hyperdr1ve · July 24, 2015, 7:12pm

I can’t seem to get traction on this bug. Here is the smallest C++ project I could make that replicates the issue.

hyperdr1ve · August 11, 2015, 5:44pm

Manually adding missing content via copy/paste works for windows but not for mobile.

JellyKnee · September 9, 2015, 8:36pm

I’m with you, packaging is broken in UE4. It’s fine for anything that’s referenced; anything else isn’t included.

You can see that simply by looking at the command line the editor passes to UAT (it starts with BuildCookRun) when you start packaging your project. It won’t pass along any extra directories specified in the .ini file.

The only workaround I have is to have a build script that removes content I don’t want packaged and then to specify that I want it to package all content (which uses the argument -CookAll). I’ve looked through the parameters supported by UAT and I can’t see one that would cook and/or package other content.

hyperdr1ve · September 9, 2015, 8:58pm

Please upvote the question if you can. I think packaging C++ projects falls into the < 1% of users bin, so I’m guessing this isn’t even on the radar. Even a recommended workaround of how to make sure a texture asset is referenced would be ok with me.

Your workaround seems doable, but very heavy-handed and difficult to maintain.

JellyKnee · September 9, 2015, 9:29pm

Oh, it’s ridiculous to do it that way, I agree. In our case, though, it takes our packaged builds from 2.75GB to 1.3GB - so it’s worth maintaining for now. But I wouldn’t have to do that if I could specify the maps to be packaged properly using the editor.

The wiki (A new, community-hosted Unreal Engine Wiki - Announcements - Unreal Engine Forums) suggests that I can add maps under the [AllMaps] section of DefaultEditor.ini. But then why does the editor add MapsToCook under [/Script/UnrealEd.ProjectPackagingSettings]? It seems like all of the array options under the Packaging UI in the editor are just thrown away when packaging - only the binary on/off options are used.

I actually don’t believe this problem is specific to C++ projects, it’s specific to any project that doesn’t want to package everything… or one that wants to package things that aren’t directly referenced by maps!

I guess I could look into fixing it myself since I’ve already done a lot of investigating - the problem is finding time with deadlines looming! I’m sure Epic has the same issue too - it’s just frustrating because in all other ways the packaging process is very good!

hyperdr1ve · September 9, 2015, 9:38pm

yep I agree, all of the array options seem to be no-ops. I’m not sure how to create unreferenced content using only blueprints, but if there were an easy repro it might help the cause.

I know what you mean about fixing it yourself. It’s on my todo list to look at the packaging code, if you ever want to collaborate on this, maybe we can share notes! I haven’t yet upgraded to 4.9 by the way, so unsure of changes there.

JellyKnee · September 9, 2015, 9:51pm

I’m on 4.9 - it still doesn’t work.

As for working on it together - I can tell you that we have to start with Engine\Source\Programs\AutomationTool\Scripts\BuildCookRun.Automation.cs - all the magic happens inside DoBuildCookRun(). I’ll definitely get around to going further down the rabbit hole, I just don’t have any time this week.

JellyKnee · September 9, 2015, 9:59pm

Actually, I went further down the rabbit hole. It looks like the problem might be inside Engine\Source\Editor\UnrealEd\Private\Commandlets\CookCommandlet.cpp. In 4.9 at least, the line in question starts at 1146.

I’ll let you know what I find out!

hyperdr1ve · September 9, 2015, 10:00pm

I’m going to pull down master tonight and dig around.

JellyKnee · September 9, 2015, 11:51pm

I don’t have time to continue looking today but what I’ve found is that where the settings are being ignored starts in SetupParams() in BuildCookRun.Automation.cs. It creates a new ProjectParams object in there - that’s the object that needs to contain all of the correct data.

In theory passing the “cookdir” parameter with directories delimited by the ‘+’ character should solve your problem! So far that seems to be specified in CookCommandlet.cpp and CommandletUtils.cs - so either they’re not working properly or it needs to be specified somewhere else too!

Hopefully that gives you something to start working with!

hyperdr1ve · September 10, 2015, 5:55pm

I spent a good 2 hours familiarizing myself with the code. I’m currently investigating the CreateStagingManifest() function in CopyBuildToStagingDirectory.Automation.cs and determining why it isn’t honoring the UFS and Non-UFS parameters in the packaging settings.

anonymous_user_33e74471 · September 11, 2015, 4:05pm

edit: Top editing because I found something useful.

Someone came across this. Looks like unreal doesn’t like to package directories with “xbox” or “ps4” in their names. Maybe try changing the directory name and see if it packages?

end edit:

I believe all the asset directories that get cooked get compiled into an intermediate file format that lives under Your_Cooked_Project_Path\Content\Paks as .pak files and the directory structure is not preserved. If you want the directory structure and all files copied across it should go under “Additional Non-Asset Directories to Copy” or Cook depending on what you want.

The reason it probably doesn’t work in the cooked build is probably because your paths in your constructor aren’t asset references, which should look like this: “Texture2D’/Game/Textures/Light01.Light01’”

I think this is more that you took a really roundabout way of loading your assets that happened to work in the editor by some random quirk of the FObjectFinder. You can copy and paste the actual asset paths by right clicking the asset in the content browser and “Copy Reference” then paste that in for your strings. Try that and see what happens.

edit: Hrm. I think the first bit I said is because we have “Create compressed cooked packages” ticked. hrmrmrm. I’ll poke around more, but we’ve had no issues using “Additional Asset Directories to Cook”.

hyperdr1ve · September 11, 2015, 5:45pm

Well, I finally found the issue. It wasn’t a bug, just an unexpected feature of the staging system.

In CopyBuildToStagingDirectory.Automation.cs, a function is called to copy UFS and NonUFS folders to the staging folder (SC.StageFiles(...)):

The final argument passed to the function is a bool called bStripFilesForOtherPlatforms that is determined using:

!Params.UsePak(SC.StageTargetPlatform)

So basically if you aren’t using a pakfile, bStripFilesForOtherPlatforms will be true and you’re in danger of getting the same behavior I describe.

public bool UsePak(Platform PlatformToCheck)
{
    return Pak || PlatformToCheck.RequiresPak(this) == Platform.PakType.Always;
}

When this boolean is set, there’s some code in DeploymentContext.cs::StageFiles that looks at the full path of the content you are including. If this content contains a folder whose name matches any platforms that you are not building for then the content gets ignored - as in not copied.

So for instance trying to include content named /Game/Content/UI/XboxOne/button.uasset when building for Win32 will get ignored because XboxOne is a platform name. If the folder name was XboxOne1 or MyXboxOne it would have been ok since the match string is “/Platform/”.

I looked through the docs and couldn’t find any mention of platform specific folder names as part of the packaging process! Kind of a huge deal here I think.

For future generations, strategically use platform names in your build folders:
Win32
Win64
WinRT
WinRT_ARM
UWP
Mac
XboxOne
PS4
IOS
Android
HTML5
Linux
AllDesktop

In short, renaming my folder from XboxOne to Xbone fixed the problem.

zerofiftyone · June 8, 2016, 3:57pm

I’ve been having this same issue for the past two months. Thank you SO MUCH for getting to the bottom of it and documenting it in this thread.

HazzaBui · June 12, 2016, 12:12am

Thanks for this answer, it helped me as well!
For anyone interested, this bug (anything which acts in an unexpected way, and is undocumented, is a bug in my eyes), still exists in 4.11.2