AssetDataTagMap.cpp InitializeFromBuilder Buffer Overflow

Development Pipeline & Plugins
, ,
1

Hey folks,

We’ve encountered an issue where the InitializeFromBuilder function in AssetDataTagMap.cpp will cause a crash when initiating a cook. The FMemoryWriter in that function is running out of memory for us. The fatal error we see is logged once we try to allocate more bytes to its internal array, beyond the max 2GB size.

I have been able to get past that line by converting the FMemoryWriter to FMemoryWriter64 but I’m somewhat unsure what to do for the FMemoryReader step immediately after, as it doesn’t seem to be able to accept a TArray of type <uint8, FDefaultAllocator64>.

Is it the right approach to change that MemoryWriter to be the 64 variant? Do you have any tips for getting past the FMemoryReader line? And lastly, is this highlighting a potential bigger issue with our project?

I’m probably unable to provide a sample project with this issue, apologies for that. I believe that this is related to the scale of our project and number of assets.

Many thanks,

David

[Attachment Removed]

2

AssetDataTagMap is used when loading the AssetRegistryCache (in <Project>\Intermediate\CachedAssetRegistry_*.bin).

I don’t know why your CachedAssetRegistry_*.bin files would be more than 2GB in size; they are shareded for every 250k assets, this limit is set in

AssetDataGathererPrivate.h:

static constexpr int32 CacheShardAssetCount = 250000;Maybe you have some assets that have more AssetRegistry data (mostly AssetDataTags) than we have encountered locally. The biggest tags we have locally are the FindInBlueprint tags which can be multiple megabytes for large blueprint packages, but even with those tags we haven’t hit the 2GB limit on a shard in Fortnite; our shards are around 400MB in size.

The limit is technically unnecessary, other than performance of editor load it shouldn’t be a problem to remove it.

Speaking of editor load, I don’t know why this would only occur for you when cooking. The AssetRegistry cache should be used when loading the editor as well. Maybe the cache is corrupted some how, rather than being a legitimately large size?

I think not having FMemoryReader64 is an oversight; it either has never come up that we need to read from 2GB+ memory buffers using an archive, or we were using another method to do it. The other method we might be using is an FMemoryReaderView, which takes an FMemoryView rather than a TArray. This code works for me locally (I didn’t test reading a 2GB+ file with it though):

	TRefCountPtr<const FStore> InitializeFromBuilder(FStoreBuilder& InBuilder)
	{
		TArray64<uint8> Data;
		FMemoryWriter64 MemoryWriter(Data, true);
		FStoreData StoreData = InBuilder.Finalize();
 
		SaveStore(StoreData, MemoryWriter);
 
		FMemoryView DataView(Data);
		FMemoryReaderView MemoryReader(DataView, true);
		return LoadStore(MemoryReader);
	}

Let me know if that works for you ; I’ll submit that change into 5.8 to remove this unnecessary restriction.

[Attachment Removed]

3

My mistake, this code is called when loading the previous DevelopmentAssetRegistry.bin file during an iterative cook, not when loading the editor’s AssetRegistry cache. That explains why it is more likely to occur (that file is not sharded) and also why it does not occur during your editor loads.

Our DevelopmentAssetRegistry.bin on Fortnite is 1.4GB; we probably would have hit this limit in the next year or two.

[Attachment Removed]

4

There are changes to TMemoryView in head that might explain why my version compiles.

Try

FMemoryView DataView(Data.GetData(), Data.Num());

[Attachment Removed]

5

That’s curious, I’m running a full cook in this case, and this branch doesn’t have one of those files already present. That said I haven’t managed a successful cook in 5.7 yet.

In 5.6.1 I can find one of those files that’s 101MB, but I imagine that might be missing the extra metadata added in 5.7 which is potentially ballooning it for us.

Unfortunately that snippet doesn’t compile for me. I’m still on the 14.38/17.8 compiler on MSVC, are you on the newer recommended-for-5.7 version?

error C2665: 'TMemoryView<const void>::TMemoryView': no overloaded function could convert all the argument typesThat error occurs on the ‘FMemoryView DataView(Data);’ line.

Many thanks,

David

[Attachment Removed]

6

I can get the following to compile but then that fires a check and the log reports a corrupted header

TArray64<uint8> Data;
FMemoryWriter64 MemoryWriter(Data, true);
FStoreData StoreData = InBuilder.Finalize();
 
SaveStore(StoreData, MemoryWriter);
 
TArrayView64<uint8> DataView(Data);
FMemoryReaderView MemoryReader(DataView, true);
return LoadStore(MemoryReader);

The check in question:

		// Unlike the other constructor, we don't need to check(InCount >= 0), because it's coming from a TArrayView which guarantees that
		if constexpr (sizeof(InCountType) > sizeof(SizeType) || (sizeof(InCountType) == sizeof(SizeType) && std::is_unsigned_v<InCountType>))
		{
			check(InCount <= static_cast<InCountType>(TNumericLimits<SizeType>::Max()));
		}

Will keep hacking in the meantime

[Attachment Removed]

7

That seems to have done the trick. Thank you kindly Matt :blush:

[Attachment Removed]