Libcurl FTP-Download incomplete

Development Getting Started & Setup
, , , ,
1

I am currently trying to download a File from a public git-Repository using curl in my Unreal C++ Project. Here is the code I’m trying to execute that I derived from the FTP-Example:

// This is in the .h file
struct FFtpFile {
	FILE* File;
	const char* Filename;
};

void FtpFetch(const std::string URL, const char* Filename) {
	CURL* Curl = curl_easy_init();

	const FFtpFile FtpFile {
		nullptr,
		Filename
	};
	
	if (!Curl) {
		UE_LOG(LogTemp, Warning, TEXT("Error Initiating cURL"));
		return;
	}

	curl_easy_setopt(Curl, CURLOPT_URL, URL.c_str());
	curl_easy_setopt(Curl, CURLOPT_VERBOSE, 1);
	curl_easy_setopt(Curl, CURLOPT_FOLLOWLOCATION, 1);
	curl_easy_setopt(Curl, CURLOPT_SSL_VERIFYPEER, false);
	
	// Data Callback
	const auto WriteCallback = +[](void* Contents, const size_t Size, const size_t NumMem, FFtpFile* FileStruct) -> size_t {
		if (!FileStruct->File) {
			fopen_s(&FileStruct->File, FileStruct->Filename, "wb");
			if (!FileStruct->File) {
				return CURLE_WRITE_ERROR;
			}
		}
		return fwrite(Contents, Size, NumMem, FileStruct->File);
	};
	curl_easy_setopt(Curl, CURLOPT_WRITEFUNCTION, WriteCallback);
	curl_easy_setopt(Curl, CURLOPT_WRITEDATA, FtpFile);

	
	const CURLcode Result = curl_easy_perform(Curl);
	if (Result != CURLE_OK) {
		const FString Message(curl_easy_strerror(Result));
		UE_LOG(LogTemp, Warning, TEXT("Error Getting Content of the Model File: %s"), *Message);
		return;
	}
	curl_easy_cleanup(Curl);

	// Close the Stream after Cleanup
	UE_LOG(LogTemp, Log, TEXT("Successfully Fetched FTP-File. Closing Write Stream"))
	if (FtpFile.File) {
		fclose(FtpFile.File);
	}
}

Note that this is executed on a separate Thread using the Unreal Async function:

void AsyncFetchModelFile(const FDatasetInformation Information, const std::string URL, FProgressCallbackEvent ProgressCallback, UGameInstance* GameInstance) {
	std::string Path = TCHAR_TO_UTF8(*UWorkspaceFunctions::AddDatasetToWorkspace(GameInstance, Information));
	
	TFunction<void()> Task = [Path, URL, ProgressCallback]() {
		FtpFetch(URL, ProgressCallback, Path.c_str());
	};

	UE_LOG(LogTemp, Log, TEXT("Fetching FTP on Background Thread"))
	Async(EAsyncExecution::Thread, Task, [](){UE_LOG(LogTemp, Warning, TEXT("Finishied FTP on Background Thread!"))});
}

The curl_global_init() and _cleanup() calls are already called outside of the function. Running the code on the main Thread doesn’t fix the problem.

To the error itself: The download runs almost flawlessly, but the downloaded file (in this case a .fbx file) always misses the last ~800 Bytes and is therefore incomplete. Also, the file keeps being open in Unreal, so I can’t delete/move the file unless I close the Editor.

Before writing this Unreal code I tried running the same code in a pure C++ setting and there it worked flawlessly. But for some reason doing the same in Unreal doesn’t work.

I also tried using a private method instead of the lambda-Function, but that didn’t make any difference.

Any help would be appreciated!

2

I didn’t manage to get it working in the end (I suspect it’s an Unreal-Related Bug), but I found another way using the included Unreal HTTP Module:

FString Path = ...

// Create the Callback when the HTTP-Request has finished
auto OnRequestComplete = [Path](FHttpRequestPtr Request, FHttpResponsePtr Response, bool bWasSuccessful) {
	if (bWasSuccessful) {
		FFileHelper::SaveArrayToFile(Response->GetContent(), *Path);
		UE_LOG(LogTemp, Log, TEXT("Successfully downloaded the file to '%s'"), *Path)
	} else {
		UE_LOG(LogTemp, Warning, TEXT("Error downloading the file (See EHttpResponseCodes): %s"), Response->GetResponseCode())
	}
};


// Create a HTTP-Request and Fetch the file
TSharedRef<IHttpRequest, ESPMode::ThreadSafe> Request = FHttpModule::Get().CreateRequest();
Request->SetVerb("GET");
Request->SetURL(URL);
Request->OnProcessRequestComplete().BindLambda(OnRequestComplete);

Request->ProcessRequest();