GPU/System complete lockup when editor tries to render tooltips in property lists [4.14 Windows]

Development Platform & Builds
, , , ,
1

So, apparently with property lists on the right side of the screen in many editor modes, such as the Static Mesh editor or Blueprint editor, if you move your mouse too fast over too many different options that display tooltips, the editor crashes the GPU. As in, the system completely locks up, and Windows has to do an emergency driver reset with the “your display driver stopped responding” balloon afterwards. While a driver crash/lockup might sound like a driver issue, it’s not. I’ve tried 3 different NVIDIA drivers (353, 375, 376) and they all have the same problem. I’ve looked at the log and there doesn’t appear to be anything useful to track it down. It’s just running normally, then all of a sudden FATAL: D3D device disappeared (or something similar) No reason specified. I’ve sent about 15-20 crash reports.

Yes, this happens with brand new projects, but not with just anything in the window. I haven’t figured out what exactly the issue is, but it never happens in the main editor window. It happens in the blueprint and static mesh editors (probably more, but those are the only two I’ve tested). And not just any mesh. In the static mesh editor, it won’t crash if the mesh is really simple or has few materials/LODs. In the blueprint editor, it seems to crash easiest if you’ve selected the blueprint’s mesh component (if it has one). But these don’t CAUSE the crash. They’re just criteria for the crash to occur. It doesn’t lock up until scrolling/mousing over a few different controls and the engine I think is trying to render tooltips. My mouse always freezes right over an option, at about the length of time it takes for a tooltip to appear. Total lockup, 10 seconds later my monitor goes black, followed by re-initialization of the GPU, the warning balloon, and the crash window for UE4.

It’s unfortunate, because 4.14 is otherwise my favorite iteration of this engine, since it fixes a lot of the bugs of 4.13 while improving on convenience, but GPU lockup makes this unusable. I’ll provide as much information as you need when I get the chance.

Also, to add, this issue was NOT present in any of the 4.14 preview releases, nor any of the 4.14 source builds I used to compile (I stopped building from source because it just took up way way too much space).

EDIT: I have found a way to reproduce this reliably, with all engine configs at their default.

  1. Start a new project with the Vehicle Advanced Blueprint template.
  2. Open the VehicleBlueprint blueprint class.
    3. If the editor opens a separate window for this, move the tab to the main window. (It only seems to happen when the BP editor is in the same window.)
  3. Select the Vehicle’s Skeletal Mesh component in the components tab.
    5. Grab and move the blueprint graph a bit with the right mouse button. (Doesn’t matter how but this needs to happen for some reason).
    6. Under the mesh heading in the details panel on the right, select the mesh’s name next to the thumbnail to bring up the drop-down list, then click off the list to close it again.
  4. Now, keeping the mouse cursor in the details panel, scroll with the mouse wheel up and down a couple times, move the mouse over random things and pause, then quickly move over other things. Hovering over the thumbnail for the Element 0 material is a pretty reliable way to trigger this.

Notes:

  • This does NOT occur when using opengl mode by starting the editor with the -opengl4 switch, no matter what I do. It’s D3D specific.
  • While this has 7 very specific steps, all of these steps are very normal things to do within an editing session at some point. Also, it’s not limited to this situation. Like I said, it also is triggered by the static mesh editor, and probably others. (In some cases, if the SM editor is loaded on startup with the project, it will IMMEDIATELY crash the GPU without even doing anything but moving the mouse just a hair.)

Nvidia GT 555M w/ Optimus (using external monitor)
Windows 7
Dell XPS L702x
Intel Core i7-2760QM

link text

2

Hello Tate Richardson,

The first thing that comes to mind is that this could be related to Optimus. We’ve had issues in the past with tooltips and context menus not rendering properly when GPU switching is being used, although I’ve never seen it cause a crash. Could you try disabling Optimus and setting it to default to the Nvidia GPU temporarily to see if the issue still occurs?

Also, since you mentioned that this is only a 4.14 release issue, have you tried 4.13 since you started seeing this issue to ensure that it is only happening in 4.14?

I’ll see if I can track down a laptop in our offices that is similar to what you’re using to try to reproduce this.

3

Sorry for not being able to determine the exact problem of yours, but in general when your system is getting locked up it’s due to the thermal sensor sending a critical overload signal to the notebook core system (on the sidenote, same applies to the desktops as well). You might want to track your performance with something like HWmonitor to figure if this is the issue. Normally it’s not a big deal to fix the hardware problem causing the overheating, you just have to be careful with cleaning, or refreshing the thermal paste.

4

Currently I’ve been running UE4 on an external monitor connected via HDMI. The way Optimus is set up, at least on mine, everything displayed on an external monitor is rendered by the Nvidia GPU, even those apps which are locked by the configuration tool to use the internal card. So it’s nothing to do with switching GPUs

Also, I did indeed have cooling problems in the past, but very recently I did a full clean of the fan and heat sink, and applied new thermal paste, and it’s currently running at temps near the same as when I first bought it. So it’s not temperature, either. Regardless, I have an app that I have programmed to switch to a different power scheme the instant it hits a certain temperature, and neither said app nor the switching has ever once caused a lockup.

Yes, I’ve tried 4.13 since then. In fact, that’s what I had to go back to to get anything done! No problems with 4.13. Also, like I said at the end of my first post, I didn’t even have this problem in the preview 4.14 releases, nor did I have this problem when I would build 4.14 from source. But I have not built from source since the official 4.14 release. I’m reluctant to do it again, because it’s a pain and takes up so much space, (of which I am now limited again since my external HDD leapt off the table to its doom), but I can try if it would help.

5

I just upgraded to 4.14.1 and this issue still persists. Happened almost immediately when I selected the UFO actor (from the flying demo), went into the blueprint, clicked on the mesh component, and started scrolling up and down. Complete lockup, screen black 8 seconds later, and driver reset with the warning.

6

Also, I had upgraded my NVIDIA drivers since the comment last month and it still does the same thing.

7

I can do anything else in the editor with MUCH more going on on screen and not have this problem. I can run the Shootergame demo on cinematic at 1900x1080 rez and I don’t overheat. Also, whenever I used to overheat, it would cause a blue screen. This isn’t a blue screen. It’s the failsafe driver reset that windows does.

8

Hello Tate,

Thank you for all of the information you’ve been able to provide. I likely should’ve asked for this earlier, but could you provide the callstack for the crash that you’re getting? From looking at the reproduction steps and your hardware, I believe that this may be related to the crash (if not the exact same as) we recently released a QFE for. Could you try the latest QFE posted on this forum thread that is listed as “Disable timestamp queries on pre-Maxwell nvidia hardware.”? This should be able to fix the issue you’re seeing.

9

Here’s what the crash reporter gave me. I haven’t installed the QFE yet cause I wanted to do one more for the stack dump.

MachineId:A0800AC84C811000C7129C8C12B69CEB
EpicAccountId:47cc8c2659f6423ab0583c78e7976e3d

Fatal error: [File:D:\Build++UE4+Release-4.14+Compile\Sync\Engine\Source\Runtime\Windows\D3D11RHI\Private\D3D11Util.cpp] [Line: 176]
Unreal Engine is exiting due to D3D device being lost. (Error: 0x887A0006 - ‘HUNG’)

UE4Editor_Core!FDebug::AssertFailed() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\core\private\misc\assertionmacros.cpp:332]
UE4Editor_D3D11RHI!TerminateOnDeviceRemoved() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\windows\d3d11rhi\private\d3d11util.cpp:176]
UE4Editor_D3D11RHI!VerifyD3D11Result() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\windows\d3d11rhi\private\d3d11util.cpp:225]
UE4Editor_D3D11RHI!FD3D11DynamicRHI::GetQueryData() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\windows\d3d11rhi\private\d3d11query.cpp:135]
UE4Editor_D3D11RHI!FD3D11DynamicRHI::RHIEndDrawingViewport() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\windows\d3d11rhi\private\d3d11viewport.cpp:570]
UE4Editor_RHI!FRHICommandListExecutor::ExecuteInner_DoExecute() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\rhi\private\rhicommandlist.cpp:250]
UE4Editor_RHI!FRHICommandListExecutor::ExecuteInner() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\rhi\private\rhicommandlist.cpp:457]
UE4Editor_RHI!FRHICommandListExecutor::ExecuteList() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\rhi\private\rhicommandlist.cpp:505]
UE4Editor_RHI!FRHICommandList::EndDrawingViewport() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\rhi\private\rhicommandlist.cpp:1359]
UE4Editor_SlateRHIRenderer!FSlateRHIRenderer::DrawWindow_RenderThread() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\slaterhirenderer\private\slaterhirenderer.cpp:486]
UE4Editor_SlateRHIRenderer!TGraphTask<FSlateRHIRenderer::DrawWindows_Private'::35’::EURCMacro_SlateDrawWindowsCommand>::ExecuteTask() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\core\public\async\taskgraphinterfaces.h:868]
UE4Editor_Core!FNamedTaskThread::ProcessTasksNamedThread() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\core\private\async\taskgraph.cpp:932]
UE4Editor_Core!FNamedTaskThread::ProcessTasksUntilQuit() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\core\private\async\taskgraph.cpp:679]
UE4Editor_RenderCore!RenderingThreadMain() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\rendercore\private\renderingthread.cpp:320]
UE4Editor_RenderCore!FRenderingThread::Run() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\rendercore\private\renderingthread.cpp:454]
UE4Editor_Core!FRunnableThreadWin::Run() [d:\build++ue4+release-4.14+compile\sync\engine\source\runtime\core\private\windows\windowsrunnablethread.cpp:74]

10

Negative. Unfortunately, the QFE did nothing for this problem. Still total lockup after scrolling up and down in the blueprint details panel with the vehicle mesh component selected in the vehicle advanced project. And presumably lots of others, since I discovered this thing on the Shootergame.

11

Thank you for that information and for giving the QFE a try. The crash you’re getting is the same one that its meant to fix but its good to know that it isn’t helping in every case and will need some additional work. I’ll provide that information to the ones in charge of the fix although we likely won’t see anything until after the holiday break unfortunately.

Edit: There should also be a more robust version of this fix included in 4.14.2 so look forward to that possibly fixing the issue as well.

12

Hello Tate,

Have you had a chance to try out 4.14.2 to see if you continue experiencing these issues? If so, please let me know so we can continue to investigate. We’re trying to get a handle on how many people the fix did/didn’t work for.

13

I have same crashes (video driver crash) at Persona (animation asset editor, animation blueprint editor):

link text

My hardware is: NVidia GTX 460.
Windows 10, driver version is 376.33

It is similar to:
Fixed! UE-38818 [CrashReport] UE4Editor_D3D11RHI!TerminateOnDeviceRemoved() [d3d11util.cpp:176]

14

Hello Tate and v.s.,

Have either of you been able to test out 4.14.3? Are you still experiencing these crashes? The hotfix should have the necessary fixes but we are seeing that some users are still experiencing this crash so it would be helpful to know if you are among those still affected.

15

Yes. It once again did the same thing. For what it’s worth, I’ve since gotten a new external HDD, so I’m able to play with the source code again. Version 4.16, the “promoted” branch as of ‎January ‎04, ‎2017, still shows this bug as well. It’s possible that the fix, if it exists, simply hadn’t been merged in to that branch yet, though. That said… as far as my experience goes, the fix doesn’t exist yet.

16

Hi Tate,

There are several things I would like you to try to see if it improves stability for your system.

The GPU crash may be related to microsoft’s Timeout Detection & Recovery(TDR) feature. You can find out more about it here: Timeout detection and recovery (TDR) - Windows drivers | Microsoft Learn

Basically, it detects if your GPU is hung up and if it passes a certain amount of time Windows will automatically reset the GPU resulting in the crash. This is done to prevent a freeze forcing a hard reboot of the system.

Sometimes you need to add a little more time for your GPU to process before Windows causes the reset. To do this you need to change the TDR delay.

I have two suggestion on how to do this you can do it manually yourself OR you can download a tool that will do it for you. However, I haven’t used the tool that much myself but it does appear to work. (Use at your own risk)

Method 1 (manually):

  • Go to the this link:Testing and debugging TDR during driver development - Windows drivers | Microsoft Learn

  • There you will find a section called TdrDelay it gives you all the information you need to create a registry key that will change the delay time

  • To open your registry open your start menu and type “run” once the window is open type “regedit” and hit OK

  • Now you should have Registry Editor open this is where you will use the info at the website I just linked.

  • First navigate the key path HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\GraphicsDrivers

  • Right click on GraphicsDrivers and go to new> DWORD (32 bit) value

  • You will see a new key appear make sure to name it “TdrDelay”

  • Right click it and select “modify”

  • Under Value data enter 10 (this is the number of seconds to delay) make sure you have Decimal selected as the base

  • Hit okay and restart your system

Method 2

A user (IronicParadox) suggested that going into the content browser pane > click the little eye that says “View Options” and uncheck “Real-Time Thumbnails.” made the engine much more stable.

please be sure to let me know if any of these workarounds help.

Ed

17

This was one of the first things I tried. The only thing increasing this number did was force me to wait longer once my screen froze. I’d set it as high as 30, and I ended up staring at a frozen, unresponsive monitor for 30 seconds. The thing about this bug is, once it hits, the mouse stops moving and there’s no animation or anything, even if I’m in mid-scroll. Complete screen lock, basically. And this would come at an instance where there shouldn’t be any heavy load on the system (or at least no heavier than the previous minutes/seconds when it was working perfectly fine). All I’m doing is scrolling the details menu up and down and hovering over items to display their tooltips. I’m not even sure if it’s related to tooltips, because usually the item I think is the offending tooltip has already been loaded and shown to me once or twice.

And yes, this is still present in 4.15. =( Sorry for taking so long to respond, but I had things going on that forced my attention away from ue4 and game dev in general for awhile.

18

This issue has gotten logged in our tracker here: Unreal Engine Issues and Bug Tracker (UE-42280)

Look at the ticket for a small list of workarounds that may help the issue.

If you want a quick and easy workaround that seemed to help a lot of people do the following.

  • Open the editor
  • In the the Content Browser select View Options (eye icon bottom right)
  • Uncheck Real Time Thumbnails

This was proposed by a user (IronicParadox).

Cheers,

Ed