Http If-None-Match logic isn't working on Mac due to the implementation of IHttpRequest with NSMutableURLRequest

Development Programming & Scripting
1

Http If-None-Match logic isn’t working on Mac due to the implementation of IHttpRequest with NSMutableURLRequest - which keeps its own internal cache by default and doesn’t honor the user’s request headers. It would need a cachePolicy of NSURLRequestReloadIgnoringLocalCacheData to disable the internal caching but the Engine doesn’t specify that. The makes the behavior of IHttpRequest on Mac inconsistent with other platforms.

Here’s the patch to the Engine to fix it: (AppleHttp.cpp, line 14):

FAppleHttpRequest::FAppleHttpRequest() 
:   Connection(nullptr)
,   CompletionStatus(EHttpRequestStatus::NotStarted)
,   ProgressBytesSent(0)
,   StartRequestTime(0.0)
,   ElapsedTime(0.0f)
{
    UE_LOG(LogHttp, Verbose, TEXT("FAppleHttpRequest::FAppleHttpRequest()"));
    Request = [[NSMutableURLRequest alloc] init];      
    Request.timeoutInterval = FHttpModule::Get().GetHttpTimeout();
    // Turn off NSURLRequest internal caching
    Request.cachePolicy = NSURLRequestReloadIgnoringLocalCacheData;
}
2

Hey iktomi,

Can you give me an example as to how now having the Request.cachePolicy line makes the If-None-Match work, as compared to now?

I need to have a test case for a bug report.

Thanks.

3

Hi,
Without that change a 304 is never returned on Mac. A test case would be to set the if-none-match header with an etag for an existing file. You’d expect a 304 and you’d get one on other platforms but on Mac you’d get 200 and the file “downloaded” (but it would actually be coming from the apple API’s internal cache).

4

Hey iktomi,

To begin, I am testing in 4.13 Release without your change.

So I put together 2 projects, one on Mac one and PC, so I could see if there was any differences. This is my result:

First, this was the test URL that has a ETag that will return 304 if If-None-Match header is set. As a note, if it is not set, normal 200 returns. [https://httpbin.org/cache][1]

Secondly, I setup the project to be ready for Http requests, which means adding the following the Game.Build.cs:

PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore", "Http" )};

Then, I created a Actor I can call to send the request:

[.h]

#pragma once

#include "GameFramework/Actor.h"
#include "Runtime/Online/HTTP/Public/Http.h"
#include "HTTPActor.generated.h"

UCLASS()
class AH492514_API AHTTPActor : public AActor
{
	GENERATED_BODY()
	
public:	
	AHTTPActor();
	virtual void BeginPlay() override;
	virtual void Tick( float DeltaSeconds ) override;

    UFUNCTION( BlueprintCallable, Category = "HTTP" )
    void CallHttp( const FString URL );

    void OnResponseRecieved( FHttpRequestPtr Request, FHttpResponsePtr Response, bool bSuccessful  );
	
    FHttpModule* Http;
};

[.cpp]

#include "AH492514.h"
#include "HTTPActor.h"

AHTTPActor::AHTTPActor()
{
	PrimaryActorTick.bCanEverTick = true;

    Http = &FHttpModule::Get( );
}

void AHTTPActor::BeginPlay()
{
	Super::BeginPlay();
	
}

void AHTTPActor::Tick( float DeltaTime )
{
	Super::Tick( DeltaTime );

}

void AHTTPActor::CallHttp( const FString URL )
{
    TSharedRef<IHttpRequest> Request = Http->CreateRequest( );
	Request->OnProcessRequestComplete( ).BindUObject( this, &AHTTPActor::OnResponseRecieved );
	
	Request->SetURL( URL );
	Request->SetVerb( "GET" );
	Request->SetHeader( TEXT("User-Agent"), "X-UnrealEngine-Agent" );
	Request->SetHeader( "Content-Type", TEXT( "application/json" ) );
	Request->SetHeader( TEXT( "If-None-Match" ), "ETag" );
	Request->ProcessRequest( );
}

void AHTTPActor::OnResponseRecieved( FHttpRequestPtr Request, FHttpResponsePtr Response, bool bSuccessful )
{
	TArray<FString> RequestHeaders = Request->GetAllHeaders();
	for( int i = 0; i < RequestHeaders.Num(); i++ )
	{
		UE_LOG( LogTemp, Warning, TEXT("Request Header: %s"), *RequestHeaders[ i ] );
	}

	int32 ResponseCode = Response->GetResponseCode();
	UE_LOG(LogTemp, Error, TEXT("ResponseCode: %d"), ResponseCode);

	TArray<FString> ResponseHeaders = Response->GetAllHeaders( );
	for( int j = 0; j < ResponseHeaders.Num(); j++ )
	{
		UE_LOG( LogTemp, Warning, TEXT("Response Header: %s"), *ResponseHeaders[ j ] );
	}
}

I then setup Blueprint to create the Actor and then call the CallHttp( ) function, like this:

108095-492514_httpblueprint.png
108095-492514_httpblueprint.png844×532 88.4 KB

This, results in the ProcessRequest with the SetHeaders( ) to respond with:

[PC and Mac]

LogTemp:Warning: Request Header: User-Agent: X-UnrealEngine-Agent
LogTemp:Warning: Request Header: Content-Type: application/json
LogTemp:Warning: Request Header: If-None-Match: ETag
LogTemp:Warning: Request Header: Content-Length: 0
LogTemp:Warning: Request Header: Pragma: no-cache
LogTemp:Warning: Request Header: Expect: 
**LogTemp:Error: ResponseCode: 304**
LogTemp:Warning: Response Header: Server: nginx
LogTemp:Warning: Response Header: Date: Thu, 22 Sep 2016 14:42:50 GMT
LogTemp:Warning: Response Header: Connection: keep-alive
LogTemp:Warning: Response Header: Access-Control-Allow-Origin: *
LogTemp:Warning: Response Header: Access-Control-Allow-Credentials: true

As you can see, I am not seeing the same result as what you are describing. If you give me anymore information as to your report, I can investigate further but as of now, we do not believe this to be a bug with the Unreal Engine.

5

Hi,

You’ll need to set the If-None-Match header to the value that you received for the ETag header in a previous response.

So your test should be something like:

  1. Send request for an existing file. Get the value of the “ETag” header from the response.
  2. Send another request for the same file this time setting the “If-None-Match” header to the value you got in step 1

Since the file hasn’t changed you’d expect a 304 (assuming your server supports etags).

6

Hey iktomi,

Thanks for the feedback. I was able to see the issue of the return code being different on Mac and have logged a issue for it. You can follow it here:

https://issues.unrealengine.com/issue/UE-36317

If you want, you can submit a pull request to the master branch for UE4.

Also, when you submit a bug, if you could follow these instructions:

https://answers.unrealengine.com/questions/12363/how-do-i-report-a-bug.html

This often helps aid both the person checking the issue, as well as yourself.

Thanks again.