This tutorial show you how to set up Dataprep recipes and Python scripts that allow you to process any number of incoming Datasmith files without even opening the editor.
https://dev.epicgames.com/community/learning/tutorials/0Gly/unreal-engine-developing-a-scalable-dataprep-pipeline-with-python
1 Like
@Stephen.Phillips - thanks for posting this, it’s really interesting. We’re hitting an issue doing something almost identically however we are running this as an unattended process on a build machine triggered by Jenkins.
Have you or anyone in Epic done anything similar? We are finding that this doesn’t run with -nullRHI as there seems to be a crash and running it without -nullRHI leads to:
LogWindows: Error: with error DXGI_ERROR_NOT_CURRENTLY_AVAILABLE
I followed up with David privately, but here’s some info for anyone else who finds this thread. It appears that when Jenkins runs as a Windows service, it is not getting proper access to the display adapter. So in the UE logs there is no graphics device detected, despite having top of the line graphics hardware. David’s solution was to, as part of the pipeline, cancel the Jenkins service and instead run all the same Jenkins operations via command line. In command line, the display adapter is registered properly and UE can do what it needs to do.
Ideally there is always a display adapter, virtual or physical, running on a computer so that UE will leverage the GPU.
If there’s any other public information you want to share about this issue @david.gillespie , feel free to clarify.
@Stephen.Phillips - you pretty much covered it. The assumption is that dataprep requires a display adapter in order to run, which Jenkins running as a service does not provide - hence needing to have Jenkins launched as a logged in user on the build agent. FYI - We’re running Jenkins one one machine so the build machine and agent is the same - something more elaborate might be needed if the Jenkins service is communicating to agent machines as part of a build farm etc.
1 Like