Debugging in Rider on Ubuntu Linux (gnome) causes editor or game to freeze on breakpoints, causing no mouse to appear

Development Platform & Builds Debugging, Optimization, & Profiling
, , , ,
1

Hi there,

Has anyone experienced this same kind of issue and knows a solution or workaround?

I’m trying to setup my development environment on an ubuntu linux machine using unreal editor 5.1 with rider from jetbrains.

However, when I debug my game through the editor in rider by clicking the debug button. I’ve set a breakpoint to fire whenever the (first person default project) projectile gets fired.

So every time I fire a projectile the unreal editor shows in gnome that it is crashing. I can still use my mouse and keyboard, but the mouse disappears in gnome when an application is crashing, it shows a message to either wait or force quit the Unreal Editor. During this time I can use my keyboard to resume the game by pressing F5 as keyboard shortcut, but this is highly frustrating because now I can not use my mouse cursor in rider when trying to debug my game.

Does anyone know of a potential solution?
I’m thinking that this is caused by the gnome desktop environment telling the desktop environment to stop showing the mouse cursor when an application has crashed…

This is happening on a multi monitor setup.

The first time it hits the breakpoint after launching the game the cursor does appear because the game doesn’t seem to have “crashed” according to gnome.

Is there perhaps a way to let gnome know not to “crash” unreal editor when the process is being debugged by rider?

1 Like
2

Hey! I looked into this 3 months ago, replying about it here as well for future travellers, it’s related to this gnome bug (or behaviour):

The problem is exactly as described in comment #4 in the launchpad link.

There is no fix yet (2023/03) but the gnome extension to “disable force quit or wait window” does work as a workaround. (Forum won’t let me add another URL since this is a new account).

If you can mark that bug as affecting you and maybe posting a comment to add pressure for the bug to get fixed it would be great.

1 Like
3

Hi!

Thank you for the great response, this was exactly what I was looking for. I’m hoping gnome devs can fix this bug somehow. I’ll post a comment saying I’m also affected by this bug.

Thank you for taking the time to respond to my question.

1 Like
4

I downloaded the extension you showed me, which helps in not displaying the wait or force close dialog, but my cursor is still not showing up, probably because the game captures the cursor when I fire the projectile… (Which is where my breakpoint is at) Is there a way to make the cursor escape the game when debugging? alt tabbing and pressing the super key doesn’t work either… It’s pretty weird.

2 Likes
5

Hey!

Sorry for the late reply, I didn’t have notifications enabled here. That might be related to the mouse lock mode set in the player controller. We have this on the PlayerController::BeginPlay: (using the Enhanced Input System)

	FInputModeGameAndUI InputMode;
	InputMode.SetLockMouseToViewportBehavior(EMouseLockMode::LockInFullscreen);
	InputMode.SetHideCursorDuringCapture(true);

	SetInputMode(InputMode);

For quick reference:

UENUM()
enum class EMouseLockMode : uint8
{
	/** Do not lock the mouse cursor to the viewport */
	DoNotLock,
	/** Only lock the mouse cursor to the viewport when the mouse is captured */
	LockOnCapture,
	/** Always lock the mouse cursor to the viewport */
	LockAlways,
	/** Lock the cursor if we´re in fullscreen */
	LockInFullscreen,
};

We use only lock in fullscreen because it’s easier to debug the game anyway, at least now during development. In a FPS or when shipping/debugging it might be a different situation though.

6

!!! Seamless solution - 5.3.2 tested !!!

I have found a decent workaround using .lldbinit script. It makes LLDB execute a function in the LinuxPlatform when a breakpoint is hit.

Simple (1 minute, global config)

This solution applies the hook globally (to all lldb runs). It might not be always desirable. The advanced one doesn’t. If you don’t use lldb anywhere except for unreal, this should suffice.

  1. You need to create a .lldbinit file in your home directory (~/.lldbinit) and paste the following:

    File: .lldbinit:
    target stop-hook add --one-liner "p ::UngrabAllInputImpl()"

  2. That’s it. The mouse should be released when a breakpoint is hit.

Explanation

The .lldbinit file is executed every time lldb is started (duh). It will hook to the stopping event (like a breakpoint) and call an unreal internal function - lldb command print (p) will execute the specified C++ code (with some caveats).

::UngrabAllInputImpl() is defined in one of the Unreal source files related to unix platform and should gracefully release the input without much side-effects.

Under the hood this calls some SDL functions, which in turn talk to X11 and do the actual job.

Advanced (5 minutes, Unreal-specific)

To address the globality issue I’ve prepared a slightly more complex solution to filter for Unreal presence, one that requires a custom python lldb extension.

  1. Create a python file ue_dbg_hook.py in a reasonable location. I’ve used my Rider installation directory; feel free to choose a different folder.

    File: ue_dbg_hook.py:
    import lldb

class UnrealBreakpointHook:
    def __init__(self, target, extra_args, internal_dict):
        pass

    running_unreal = None

    @staticmethod
    def check_module_loaded(debugger, module_name):
        # Get the current target
        target = debugger.GetSelectedTarget()

        # Iterate over all modules in the target
        for module in target.module_iter():
            # Check if the module name matches
            if module.GetFileSpec().GetFilename() == module_name:
                return True

        return False

    def handle_stop(self, exe_ctx, stream):
        tgt = exe_ctx.GetTarget()
        dbg = tgt.GetDebugger()

        if UnrealBreakpointHook.running_unreal is None:
            UnrealBreakpointHook.running_unreal = self.check_module_loaded(dbg, "UnrealEditor")

        if UnrealBreakpointHook.running_unreal:
            dbg.HandleCommand("p ::UngrabAllInputImpl()")
        return True


def __lldb_init_module(debugger, internal_dict):
    print("ue_dbg_hook loaded")

  2. Change the .lldbinit slightly to use the python hook instead. Don’t forget to use the path where you actually placed the python script.

    NOTE: Notice that in the second line ue_dbg_hook module name is used - if for some reason you decided to name your python script differently, remember to apply this change here as well.

    File: .lldbinit:
    command script import "~/JetBrains Rider-2023.3.2/ue_dbg_hook.py"
target stop-hook add -P ue_dbg_hook.UnrealBreakpointHook

  3. That’s it. The mouse should be released when a breakpoint is hit, but only in Unreal.

Disclaimers, notes

  • I haven’t tested this thoroughly yet, nor I have used this for an extended period of time - the solution appears to be “clean” in a sense that it doesn’t interfere with much of the engine/system, but I cannot give any guarantees for no side effects or derived issues.

  • I haven’t tested that with packaged builds either. I’m pretty sure the simple solution will work right away and for the advanced one a different module name to look for should be selected (UnrealEditor is the current choice).

  • The python hook in the advanced solution caches the unreal presence to reduce performance impact - it might not be desired when debugging low-level engine, but I honestly cannot think of any situation where one would need to break earlier. Just keep in mind that such feature exists in case some troubleshooting is in order.

  • It should work with different Unreal versions, provided the ::UngrabAllInputImpl exists. If for some reason this convenient function goes the dodo way, there is plenty of room to find a different route.

Cheers.

1 Like
7

Unfortunately, it didn’t help me. I tried both options.

Do you have any ideas how to debug this?

*Ubuntu 22.04.3 x64
GNOME 42.9
X11
UE 5.3.2

8

Looks like works for me, Thanks!

9

(post deleted by author)

10

I’ve done a lot more research. Noisycat posted a solution:

target stop-hook add --one-liner "p ::UngrabAllInputImpl()"

Unfortunately, that didn’t work for me, so I investigated further. Unreal on Linux is layered on top of SDL. There are at least two things in SDL that can capture the mouse:

  1. SDL_CaptureMouse(SDL_TRUE). Unreal calls this function in a few situations.

  2. SDL_HINT_MOUSE_AUTO_CAPTURE. This is enabled by default. Whenever you generate a mouse-down event, SDL auto-captures the mouse. If it didn’t do this, then if you clicked the mouse inside the Unreal window, then dragged the mouse to some other window, and then released the mouse, then the mouse-down event would go to the Unreal window, but the mouse-up event would go to the other window. That would suck, because Unreal would think the mouse was still down.

Noisycat’s solution calls UngrabAllInputImpl, which is supposed to uncapture the mouse. It contains a call to SDL_CaptureMouse(SDL_FALSE), which is effective only if the mouse was captured by SDL_CaptureMouse(SDL_TRUE). But if the mouse was captured by SDL_HINT_MOUSE_AUTO_CAPTURE, then SDL_CaptureMouse(SDL_FALSE) has no effect.

So, if you hit a breakpoint at a moment when the mouse is down, then the mouse is going to be in a captured state. In this situation, Noisycat’s solution fails to uncapture the mouse. In my game, a click-and-drag action was triggering the breakpoint, so it was always in a bad state in the debugger.

Honestly, I don’t have a perfect solution, but I do have some experimental ones. You can turn off mouse-auto-capture entirely by inserting the following code into FLinuxPlatformApplicationMisc::InitSDL:

SDL_SetHint(SDL_HINT_MOUSE_AUTO_CAPTURE, "0");

So far, that’s been working reliably for me, when I use it in combination with Noisycat’s solution.
That could cause problems with mouse-up events going to the wrong window. So far, that hasn’t been a problem for me. Maybe Unreal already contains code to deal with this.

Here’s a different solution that didn’t work, but I don’t know why not:

target stop-hook add --one-liner "p ::SDL_UpdateMouseCapture(SDL_TRUE)"

That routine, SDL_UpdateMouseCapture, is an internal SDL routine. If the parameter is SDL_TRUE, then that forces a mouse uncapture, regardless of whether the mouse was captured by SDL_CaptureMouse or SDL_HINT_MOUSE_AUTO_CAPTURE. In my experience so far, this seems to work some of the time, but definitely not all of the time.

In theory, I like the second solution better, because it doesn’t mess with Unreal’s normal behavior. But sadly, it didn’t work. Maybe you can continue my research and figure out how to make it work. If you do, please followup here. I’d like to know the solution.

By the way, you can put a “target stop-hook” command into lldbinit, but instead, you can put it into your code-workspace, in the launch configuration initCommands. Doing this avoids the need for Noisycat’s python wrapper.

Thanks for Noisycat for doing the research that led up to this!