[Horde] Error while cooking

Development Pipeline & Plugins
, ,
1

Heya

We sometimes get build breaks with the following error:

“Filename extracted from temp storage has different size to file in manifest.”

Since the files in question come from the Artifacts, is this caused exclusively by a network glitch?

Unfortunately, I haven’t been able to find precise repro steps, but it does happen fairly often.

It can occur on one platform but not others, or even only on certain targetConfigs — for example, happening in Shipping but not in Dev, etc.

Has anyone else run into this error?

2

Hi,

Are you able to provide more logging? is there additional error messages apart from this one?

What backend are you using for temp storage?

Matthew

3

Sorry for the delay.

Current logs about it looks like that :

Written 540 files (3142.1mb, 0.0mb/s)

Written 540 files (3142.1mb, 0.0mb/s)

Written 581 files (3162.6mb, 5.4mb/s)

Written 581 files (3162.6mb, 0.0mb/s)

Written 581 files (3162.6mb, 0.0mb/s)

Written 581 files (3162.6mb, 0.0mb/s)

Written 581 files (3162.6mb, 0.0mb/s)

Written 581 files (3162.6mb, 0.0mb/s)

Elapsed: 124s, bundle.packet_cache.hits: 1,259, bundle.packet_cache.misses: 87, bundle.bundle_cache.hits: 87, bundle.bundle_cache.misses: 2, backend.http.wall_time_secs: 0, backend.http.num_bytes: 0, bundle.packet_reader.num_bytes_read: 0, bundle.packet_reader.num_encoded_bytes_read: 0, bundle.packet_reader.num_decoded_bytes_read: 0

File Engine/Binaries/Win64/ShaderCompileWorker-Core.dll extracted from temp storage has different size to file in manifest (manifest: 13335552, local: 13335040)

It fetches the artefacts, than we get this. Note that it could be any files previously build and stored as artefacts.

It’s not 100%, it happens every now and then.

Our backend for temp storage is our Local NAS.

Is it possible that simultaneous builds could overwrite artefacts and create such issue?

4

Hello, from the information so far it does appear this could be transient error between the local NAS and Horde. Does your NAS application have any internal logs that might have additional information?

> Is it possible that simultaneous builds could overwrite artefacts and create such issue

No, horde is designed to support more than one build running at the same time when reading from temp storage.

I will open an backlog task for the team to see when this error happens if temp storage can try to fix itself for example retrying the copy.

Matthew

5

I don’t have any NAS logs to offer for now. However, I did look at the logs last time I got the issue and there was nothing outstanding other than our regular network activities.

I’ll dig deeper next time I get the issue!

It’s been more than a week without it showing its nose… which is quite surprising considering it was happening daily.

6

Thanks for the update. Because you have not seen any errors since I will mark this ticket closed for now.

Mathew