Packaging Large Arch Viz projects

Hello Everyone!

Hopefully someone can point in the right direction or at least give me some advice.

I’ve been working on a rather large project with lots of geometry and lighting. In order to achieve a high quality build everything has had it’s lightmap resolution strategically bumped up which leads to long build times (which I can deal with), but also with an extremely large file size which can’t be packaged because it’s over 2gb. I’ve tried breaking it up with level streaming, but that doesn’t really reduce the file size it just distributes it across multiple maps which helps with framerate during game, but not with final packaging. To get by I’ve had to break up the file and do small builds of each area which gives me multiple *.exe files. I’m sure its something simple, but can anyone let me know how you get around this 2gb limit and the best way to break up a large project?

Thanks in advance for any guidance!

Cheers

We’ve built projects with 10gb+ of textures before so I don’t know what this limit is.

Hmmm, this sounds wrong, I think. I never had problems with packaging big maps with like 20+GB in assets. What platform are you packaging for?

Thank you for the reply! It’s good to hear we can package larger projects. This is a snippet of the error i receive when trying to package them.

ATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: Fatal error: [File:D:\Build++UE4+Release-4.16+Compile\Sync\Engine\Source\Runtime\Core\Public\Serialization/MemoryWriter.h] [Line: 42]
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: FMemoryWriter does not support data larger than 2GB. Archive name: None.

The error occurs right at the end of the build when it’s collecting all the assets. We are trying to build to windows 64bit package. Any ideas?

Thanks again!

Is there a particular file that is over 2GB? That’s what the error sounds like.

No particular file is over 2gb. I can delete half the floor and then run a packaged project fine, which is what we had to do in order to get something out to share. Let me recreate the error and see if there is more info in the output log. At least it’s good to hear we can go over 2gb map size…i knew something seemed off with that.

Ok, so I went back to recreate the issue i was running into. Below is the output log where the packaging fails for me. All the cooking goes smoothly, it’s right at the end where it fails. Thoughts? Any assistance is greatly appreciated!

UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: begin: stack for UAT
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: === Critical error: ===
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error:
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: Fatal error: [File:D:\Build++UE4+Release-4.16+Compile\Sync\Engine\Source\Runtime\Core\Public\Serialization/MemoryWriter.h] [Line: 42]
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: FMemoryWriter does not support data larger than 2GB. Archive name: None.
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error:
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error:
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: KERNELBASE.dll!0x00000000B96C1F28
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-Core.dll!0x000000007489AFFE
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-Core.dll!0x00000000746B249B
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-Core.dll!0x0000000074640849
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-CoreUObject.dll!0x0000000089FDC2A4
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-CoreUObject.dll!0x000000008A0D39A7
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-CoreUObject.dll!0x000000008A2037FC
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-UnrealEd.dll!0x000000006FD1B7B0
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-UnrealEd.dll!0x000000006FBCC7F4
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-UnrealEd.dll!0x000000006FBD836E
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-UnrealEd.dll!0x000000006FA86E22
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-UnrealEd.dll!0x000000006FAABAD1
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-Cmd.exe!0x00000000D7247B8F
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-Cmd.exe!0x00000000D723EEDA
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-Cmd.exe!0x00000000D723F12A
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-Cmd.exe!0x00000000D7250CF9
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: UE4Editor-Cmd.exe!0x00000000D7251A1F
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: KERNEL32.DLL!0x00000000BC0A8102
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: ntdll.dll!0x00000000BCC5C5B4
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: ntdll.dll!0x00000000BCC5C5B4
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error:
UATHelper: Packaging (Windows (64-bit)): Cook: LogWindows:Error: end: stack for UAT
UATHelper: Packaging (Windows (64-bit)): CommandUtils.Run: Run: Took 124.5126995s to run UE4Editor-Cmd.exe, ExitCode=3
UATHelper: Packaging (Windows (64-bit)): Project.Cook: Cook failed. Deleting cooked data.
UATHelper: Packaging (Windows (64-bit)): Program.Main: ERROR: AutomationTool terminated with exception: AutomationTool.AutomationException: Cook failed. —> AutomationTool.CommandletException: BUILD FAILED: Failed while running Cook for C:…uproject; see log C:\Users\Mic
hael\AppData\Roaming\Unreal Engine\AutomationTool\Logs\C+Program+Files+Epic+Games+UE_4.16\Cook-2017.08.01-11.51.03.txt
UATHelper: Packaging (Windows (64-bit)): at AutomationTool.CommandUtils.RunCommandlet(FileReference ProjectName, String UE4Exe, String Commandlet, String Parameters)
UATHelper: Packaging (Windows (64-bit)): at AutomationTool.CommandUtils.CookCommandlet(FileReference ProjectName, String UE4Exe, String] Maps, String] Dirs, String InternationalizationPreset, String] CulturesToCook, String TargetPlatform, String Parameters)
UATHelper: Packaging (Windows (64-bit)): at Project.Cook(ProjectParams Params)
UATHelper: Packaging (Windows (64-bit)): — End of inner exception stack trace —
UATHelper: Packaging (Windows (64-bit)): at Project.Cook(ProjectParams Params)
UATHelper: Packaging (Windows (64-bit)): at BuildCookRun.DoBuildCookRun(ProjectParams Params)
UATHelper: Packaging (Windows (64-bit)): at BuildCookRun.ExecuteBuild()
UATHelper: Packaging (Windows (64-bit)): at AutomationTool.BuildCommand.Execute()
UATHelper: Packaging (Windows (64-bit)): at AutomationTool.Automation.Execute(List1 CommandsToExecute, CaselessDictionary1 Commands)
UATHelper: Packaging (Windows (64-bit)): at AutomationTool.Automation.Process(String] Arguments)
UATHelper: Packaging (Windows (64-bit)): at AutomationTool.Program.MainProc(Object Param)
UATHelper: Packaging (Windows (64-bit)): at AutomationTool.InternalUtils.RunSingleInstance(Func`2 Main, Object Param)
UATHelper: Packaging (Windows (64-bit)): at AutomationTool.Program.Main()
UATHelper: Packaging (Windows (64-bit)): Program.Main: AutomationTool exiting with ExitCode=25 (Error_UnknownCookFailure)
UATHelper: Packaging (Windows (64-bit)): BUILD FAILED
PackagingResults:Error: Error Unknown Cook Failure

https://www.dropbox.com/s/82i36bo2wgzxqag/Output%20Log.txt?dl=0

I strongly suggest you to make one post in answer hub https://answers.unrealengine.com/ Big chance for a staff from Epic Game to come and help you resolving the problem.

Thank you ZacD. I did see that post, but noticed it wasn’t really resolved. I’m trying to get to the root and understand the limitations when it comes to files size and packaging final build. If I split up my level using level streaming I will still end up with a collective file size well over 2gb, just spread across different maps. I’m sure it’s something simple I am missing to resolve this, just can’t figure it out.

Thanks for the suggestion, I’ll post there as well and see if I can get any more info.

So if I’m understanding correctly, you all are able to completely package a map size over 2gb without receiving an error?

Yes, I never had your problem, I have done a lot Arch viz/VR projects that are over 15gb and have worked every workday on UE4 for 2 years.

I have an idea though, try to migrate your maps/levels to a new project. Make a new blank project without any assets > save > close, open your problematic project and migrate the levels that you are using to the new project. Migration is just moving assets from one project to another. To migrate the level just select it in content browser > right click > asset actions > Migrate > pick the “Content” folder of the new project and migrate. Try packaging with the new project and see if you get the error.

Maybe the max size for one sub level is 2 GB. You could try to create sub levels. You can have as many sub levels as you want. Make sure to set the streaming to “always loaded”. Is that what you mean with level streaming? Level streaming usually means that certain levels are loaded only when you enter the area where the level is visible. But that won’t change your package size.

The easiest way to create a sub level is to select a bunch of actors, lets say all the walls and doors of the building and then in the level browser choose “create new level with selected actors”. Make sure you “save all” immediately after splitting the level otherwise you have a big mess if the editor crashes.

And you will have to rebuild lighting afterwards.