FTP-Download via libCurl

Development Programming & Scripting C++
,
1

Hey there.

I am currently trying to download a File from a public git-Repository using curl in my Unreal C++ Project. Here is the code I’m trying to execute that I derived from the FTP-Example:

// This is in the .h file
struct FFtpFile {
FILE* File;
const char* Filename;
};

void FtpFetch(const std::string URL, FProgressCallbackEvent CallbackEvent, const char* Filename) {
CURL* Curl = curl_easy_init();

const FFtpFile FtpFile {
nullptr,
Filename
};

if (!Curl) {
UE_LOG(LogTemp, Warning, TEXT(“Error Initiating cURL”));
return;
}

curl_easy_setopt(Curl, CURLOPT_URL, URL.c_str());
curl_easy_setopt(Curl, CURLOPT_VERBOSE, 1);
curl_easy_setopt(Curl, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(Curl, CURLOPT_SSL_VERIFYPEER, false);

// Data Callback
const auto WriteCallback = +(void* Contents, const size_t Size, const size_t NumMem, FFtpFile* FileStruct) → size_t {
if (!FileStruct->File) {
fopen_s(&FileStruct->File, FileStruct->Filename, “wb”);
if (!FileStruct->File) {
return CURLE_WRITE_ERROR;
}
}
return fwrite(Contents, Size, NumMem, FileStruct->File);
};
curl_easy_setopt(Curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(Curl, CURLOPT_WRITEDATA, FtpFile);

const CURLcode Result = curl_easy_perform(Curl);
if (Result != CURLE_OK) {
const FString Message(curl_easy_strerror(Result));
UE_LOG(LogTemp, Warning, TEXT(“Error Getting Content of the Model File: %s”), *Message);
return;
}
curl_easy_cleanup(Curl);

// Close the Stream after Cleanup
UE_LOG(LogTemp, Log, TEXT(“Successfully Fetched FTP-File. Closing Write Stream”))
if (FtpFile.File) {
fclose(FtpFile.File);
}
}

Note that this is executed on a separate Thread using the Unreal Async function:

void AsyncFetchModelFile(const FDatasetInformation Information, const std::string URL, FProgressCallbackEvent ProgressCallback, UGameInstance* GameInstance) {
std::string Path = …

TFunction<void()> Task = Path, URL, ProgressCallback {
FtpFetch(URL, ProgressCallback, Path.c_str());
};

UE_LOG(LogTemp, Log, TEXT(“Fetching FTP on Background Thread”))
Async(EAsyncExecution::Thread, Task, {UE_LOG(LogTemp, Warning, TEXT(“Finishied FTP on Background Thread!”))});
}

The curl_global_init() and _cleanup() functions are already called outside of this function!

To the error itself: The download runs almost flawlessly, but the downloaded file (in this case a .fbx file) always misses the last ~800 Bytes and is therefore incomplete. Also, the file keeps being open in Unreal, so I can’t delete/move the file unless I close the Editor.

Before writing this Unreal code I tried running the same code in a pure C++ setting and there it worked flawlessly. But for some reason doing the same in Unreal doesn’t work.

I also tried using a private method instead of the lambda-Function, but that didn’t make any difference.

Any help would be appreciated
~Okaghana

2

I didn’t manage to get it working in the end (I suspect it’s an Unreal-Related Bug), but I found another way using the included Unreal HTTP Module:

FString Path = ...

// Create the Callback when the HTTP-Request has finished
auto OnRequestComplete = [Path](FHttpRequestPtr Request, FHttpResponsePtr Response, bool bWasSuccessful) {
	if (bWasSuccessful) {
		FFileHelper::SaveArrayToFile(Response->GetContent(), *Path);
		UE_LOG(LogTemp, Log, TEXT("Successfully downloaded the file to '%s'"), *Path)
	} else {
		UE_LOG(LogTemp, Warning, TEXT("Error downloading the file (See EHttpResponseCodes): %s"), Response->GetResponseCode())
	}
};


// Create a HTTP-Request and Fetch the file
TSharedRef<IHttpRequest, ESPMode::ThreadSafe> Request = FHttpModule::Get().CreateRequest();
Request->SetVerb("GET");
Request->SetURL(URL);
Request->OnProcessRequestComplete().BindLambda(OnRequestComplete);

Request->ProcessRequest();
1 Like