Announcement

Collapse
No announcement yet.

FREE: Web UI (HTML/CSS/JS Interface)

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    #16
    Originally posted by One.xi View Post
    Spidershift hello again, sorry for bothering you, I thought of an idea of getiting current texture referenced in webui material and use it's pixel data to verify which pixel is completly transparent and if it is it will set the widget visibility to hit test invisible, and if the pixel from the x, y of the mouse current position will not be transparent then set the widget visibility to visible. It will run this check on every tick. What do you think of this solution to handle the "click through" transparent zones of the web ui layer? Is it viable?
    Alright so after looking more into this here is a quick prototype that I put together using blueprints. I'll obviously develop something more streamlined via C++ for the plugin itself but at least this allows us to experiment with the URetainerBox.

    If you start with the example project these are some basic changes to get things up and running. First go into the MyController blueprint and enable the following options:

    Click image for larger version  Name:	1.PNG Views:	1 Size:	25.5 KB ID:	1607364

    This provides a "clickable interface" and allows any actor to respond to the OnClicked() or OnBeginCursorOver() and OnEndCursorOver() events. Now create a temporary blueprint called MyActor and have it parented from StaticMeshActor instead of just the default Actor class. Then add the following nodes to the event graph for debugging:

    Click image for larger version  Name:	2.PNG Views:	1 Size:	99.5 KB ID:	1607365

    Once this blueprint is created you can go into MyMap and replace any static mesh actors with your custom blueprint. Right-click on each mesh you'd like to debug and select "Replace Selected Actors with":

    Click image for larger version  Name:	3.PNG Views:	1 Size:	100.3 KB ID:	1607366

    Then open the InterfaceHUD blueprint and change the default input mode from UI Only to Game and UI instead:

    Click image for larger version  Name:	7.PNG Views:	1 Size:	120.2 KB ID:	1607370

    Also open the MyHUD blueprint and change the default visibility from Visible to Self Hit Test Invisible:

    Click image for larger version  Name:	6.PNG Views:	1 Size:	59.2 KB ID:	1607371

    Now open the WebInterface user widget and wrap the Browser sub-widget with a Retainer Box.

    Click image for larger version  Name:	4.PNG Views:	1 Size:	40.1 KB ID:	1607367

    After debugging I've discovered that Hit Test Invisible is very buggy (not surprised) and doesn't always work as intended. For instance even if I have Hit Test Invisible set on the entire WebInterface widget the Retainer Box is still treated as Visible for some reason even though it's a child widget. So the Retainer Box unfortunately blocks mouse input from passing through if Visible regardless of the parent widget. Therefore the safest setup is to make everything Self Hit Test Invisible and only toggle the visibility of the Browser variable:

    Click image for larger version  Name:	5.PNG Views:	1 Size:	29.9 KB ID:	1607368

    Also not surprising is that the engine developers didn't expose the render target of a URetainerBox in any way. There's a function for GetEffectMaterial() but no function for GetRenderTarget(). Even worse they made the local variable completely private instead of protected because they like making things that much harder with their incompetence.

    Therefore the only way to get the render target is to actually have the internal mechanics of the URetainerBox set the texture parameter of your material, and then use GetTextureParameterValue() to obtain the reference. Otherwise you'd have to use a UWidgetComponent but that is more suited to having the widget in 3D world space, not 2D screen space. Moreover the URetainerBox uses the exact dimensions of the widget for the render target, so there's no extra settings to deal with in terms of dimensions.

    Since we can't directly reference the render target, make sure you use the exact same parameter name for the URetainerBox as you do in your call to GetTextureParameterValue() along with the texture parameter in the material:

    Click image for larger version  Name:	8.PNG Views:	1 Size:	83.9 KB ID:	1607369

    Note that the texture parameter matches the widget and the material type is "User Interface" along with a blend mode of "Translucent". Again we are doing this for absolutely no reason other than to obtain a reference to the render target itself. Hopefully we don't have to copy/paste too much C++ from the SRetainerWidget codebase to make this work natively in the WebUI plugin.

    At this point if you set the Browser sub-widget in your WebInterface blueprint to Hit Test Invisible you can actually click PLAY and test your "clickable interface" to see if it's working. When you move the mouse around you should see hover events for the various static mesh actors that you replaced. If you click on them you should see a click event as well.

    Now our goal is to allow these clicks to pass through behind the interface based on the opacity of the render target under the mouse cursor. It would be great if we could reroute the click events by overriding methods such as UpdateAndDispatchHitBoxClickEvents() in AHUD but as usual the short-sighted engine developers did not make that function virtual. Therefore your recommendation of toggling the visibility on each tick should at least minimize any issues with built-in click events for the time being. However the long term goal should still be to find a way to implement something natively in the engine that will intercept viewport hitboxes.

    So all we really need to do for now is read the pixel at the current mouse location on each tick and toggle the visibility of the Browser sub-widget accordingly. Go back to your MyHUD blueprint and setup something similar to this in the OnTick() event (source is below for copy/pasting directly into blueprints):

    Click image for larger version

Name:	12.PNG
Views:	56
Size:	43.4 KB
ID:	1607377

    Click image for larger version  Name:	9.PNG Views:	1 Size:	71.6 KB ID:	1607372

    Click image for larger version  Name:	10.PNG Views:	1 Size:	75.9 KB ID:	1607373

    Click image for larger version  Name:	11.PNG Views:	1 Size:	92.2 KB ID:	1607374



    Now click PLAY again and you should be able to interact with the meshes behind the interface based on the opacity of each pixel along with that 0.333 threshold in the screenshots. Keep in mind this is not an ideal solution as I've already noticed that the web page will occasionally keep buttons focused/highlighted when toggling to Hit Test Invisible. Therefore we'll probably want to manually trigger a MouseLeave event or create some kind of frame delay. This is why actually injecting into the hitbox detection of the viewport is a more appropriate solution, but for now this is at least a workable prototype.

    I will draft up something more concrete in C++ for the WebUI plugin so this can be natively supported with minimal setup. I'll let you know if I have any further updates on this. Thanks again for the suggestion, and hope this gives you a good start on having a more dynamic interface!
    Last edited by spidershift; 04-13-2019, 12:57 PM.

    Comment


      #17
      Originally posted by spidershift View Post


      As mentioned the plugin is free and on GitHub, but you will need to link your Epic Games account to your GitHub account using the instructions at unrealengine.com/ue4-on-github since the repository is private. Otherwise you will see the following 404 error page:


      spidershift´╝îthank you .

      Comment


        #18
        Hey Spidershift, I read through your implementation and was quite impressed)) You did a great job, in meantime this days I also expermented with that, but just only in c++. I've managed to get some results, Ill show you later or if you have a discord you can message me there)

        Some notes (not sure about them all)

        1. I didnt see any use of materials that where located in WebBrowserContent location. I tried to pin/unpin the nodes from the texture to material but it have no effect so I thought they are not used at all. Also no references that lead to some implications in the actual code for this materials.
        2. I did a version without the need to bind to any mouse events, just check on the tick for now ( however it's better if I could find a way to bind to a delegate inside WebInterface for OnMouseMove)
        3. To pass the click or move event down to the game layer I see only 2 solutions as you already mentioned to somehow propagete them down. Or just to toggle visible/hit test visibility on the widget itself, for now I just toggle the visibility.
        4. This way we can avoid having to deal with creating a base class for all meshes.

        I will keep you in touch.

        Again repeating myself I REALLY LOVE your plugin. It's just amazing. I can donate up do 30$ if you allow me to. Thank you very much for it.

        Comment


          #19
          Hello again spidershift, so I recorded what I managed to do. But its still a rought work, not so configurable etc.

          I am not even sure if I could do it without messing around with WebBrowserWidget itself. I am thinking of making an engine fork for my project separately. But I am still thinking..

          Anyways the result is this:





          Some code how I did it.

           
          Spoiler
          Last edited by One.xi; 04-17-2019, 03:26 PM.

          Comment


            #20
            Hello spidershift!
            We are working on a project where we use the WebUI plugin for developing the user interface with React. We are currently facing some problems and it would be nice to be able to debug our app. It's running perfectly in Chromium but when loaded in Unreal with WebUI its not working as intended.

            As far as I understand, WebUI is using the internal Chromium Embedded Framework in UE4, which should have a debugger tool that can be opened on localhost: port.

            Is it possible with WebUI to open the devTools? And if so, how do we know which port it's using?

            Thanks in advance!

            Comment


              #21
              Hi
              We are buildung a project which we are exporting for the HTML-Player but the Web-UI seems not to work there.
              Is it only for Desktop Standalone Applications and not for Browser-Applications?

              Kind regards
              Alain

              Comment


                #22
                Hey guys sorry for the delay. We've been really busy and were not able to get around to some of these updates until now. There are new versions available on GitHub and some updated documentation and examples will be posted soon. Here are some quick notes for these updates:

                WebUI
                - Native support for 60 FPS (no engine changes required)
                there is a new setting in the widget blueprint under the "behavior" section that allows you to specify the FPS for the browser instance of that widget

                - Dynamic mouse transparency
                another setting is available in the "behavior" section of the widget blueprint to enable mouse transparency which allows for clicking behind the interface

                - Debugging via the Chrome DevTools
                you can now press CTRL+SHIFT+I to open the devtools in development and debug builds for the currently focused browser widget

                - Access texture data
                new functions such as GetTextureWidth() and GetTextureHeight() along with ReadPixels() provide the ability to sample the browser pixels (this DOES NOT create a new render target and samples the existing slate texture on the render thread)

                - Focus browser elements
                new functions such as Focus() allow the browser to receive direct focus, when combined with JavaScript focus events you can now directly trigger focus into a browser input or textbox element (this was not previously possible because the browser required the player to click on the widget before receiving keyboard focus)

                - Load local files (mobile)
                you can new use functions such as LoadFile() on mobile browsers to directly access HTML files and load local image files, the file paths will be properly resolved for both Android and iOS (the updated example project will demonstrate full mobile support out of the box)

                - Custom mouse cursors
                new documentation will be provided for fixing engine-related issues that prevent custom mouse cursors from working with browser widgets

                - Support for WebGL
                new documentation will be provided for enabling WebGL in the UnrealCEFSubprocess (Windows, Mac, and Linux)

                - Support for 3D widgets (coming soon)
                the new ReadPixels() function when used with the widget interaction component allows for the 3D laser pointer to pass through transparent parts of the browser


                JSON Library
                - Direct reference checking
                you can now directly compare objects and lists via reference using the new Equals() function, subsequently new functions such as FindList() and RemoveList() have been added to the JSON array wrapper class

                - Primitive type coercion
                originally if you used GetBoolean() or GetFloat() or GetString() and the underlying JSON type did not match then the default value was returned, now these functions will automatically try to convert these values into the appropriate data type (the new equals function is now the equivalent of the JavaScript abstract equality operator)

                - Variable change tracking
                new delegate pins have been added to the Construct Object and Construct List nodes that allow for tracking variable changes (these events are only valid for the current reference and will no longer trigger if the object/array is accessed from a parent object or converted into a generic value)

                - Performance improvements
                the JSON wrapper classes now utilize the underlying FJsonValueArray and FJsonValueObject structures

                Comment


                  #23
                  Hi spidershift

                  Sorry for my stupid question.
                  But could you please tell me if WebUI is for Standalone Applications only (for example *.exe-Files on Windows) or does it also work for In-Browser-Applications (using the HTML-Canvas-Element)?

                  Kind regards
                  Alain

                  Comment


                    #24
                    Originally posted by FixundFoxi View Post
                    Hi spidershift

                    Sorry for my stupid question.
                    But could you please tell me if WebUI is for Standalone Applications only (for example *.exe-Files on Windows) or does it also work for In-Browser-Applications (using the HTML-Canvas-Element)?

                    Kind regards
                    Alain
                    I usually ignore questions that people are clearly able to find the answers to themselves but since you've asked multiple times here you go:





                    However web-based user interfaces is probably the least of your concern right now. I would be worried about the fact that Epic Games just deprecated your entire platform in the 4.23 version of the engine:

                    https://docs.unrealengine.com/en-US/...ML5/index.html
                    https://github.com/EpicGames/UnrealE...416b71021eab53

                    This is because if something doesn't benefit Fortnite or anything making them money they don't actually care at all, which is why Paragon was scrapped and now the entire H5 platform as well. It's only a matter of time before this engine reaches a breaking point. I think the CTO at my previous job put it best, which is that these engine developers are constantly adding new features without ever finishing anything they start and creating an evermore buggy and bloated product. I wouldn't be surprised if we all get to a point where we stop downloading the latest engine updates when things become too unstable. I mean they don't even bother to update their previous branches with bug fixes once they've moved on to the next version and seem to think 2-3 minor updates per engine version is enough to fix the bugs. Yet there are many times you find yourself needing to download the next major version of the engine just to fix an issue only to have new things completely broken in the new version. It's pretty apparent this engine doesn't even come close to the quality of the Source Engine and we didn't even realize what we had until it was gone. There's no question if the Source 2 Engine came out tomorrow our company would jump ship from this engine without ever thinking twice.

                    Comment


                      #25
                      @spidershift

                      Thanks for taking you time and providing us more background-informations.
                      As you may noticed, I know almost nothing about unreal engine, and the fact that it run for some big OS, from my point of view, does not automaticaly mean is does not run in a Canvas-Element of a Browser. It maybe logic from a developer-view but not for not or an ignorant one as I am :-)

                      I wish you a good day.

                      Comment


                        #26
                        Hi!

                        I am currently struggling with mobile (VR) with a 3D widget. the widget itself is the full 1920x1080 I've set it to, however the browser itself seems to cut as if it believes it is in landscape mode perhaps?

                        I am not getting the full height utilized, it's actually stretched. so the content on mobile seems wrong, it's squished in height.

                        Any ideas?

                        [SOLVED]
                        Last edited by Tumenor; 09-11-2019, 06:32 AM.

                        Comment


                          #27
                          Wow, they're completely dropping HTML5?

                          Anyway, thanks for this super useful plugin. It has made UI development fun again

                          Comment


                            #28
                            I seem to be having trouble getting range sliders to work properly. When mousing down on the slider "handle" and dragging, it may move for a short distance but then shortly stops responding. It seems like it stops processing if the mouse cursor moves off of the handle. This is on 4.22.3 editor on Windows 10, with the player controller input mode set to Input Mode Game and UI, with mouse transparency enabled on the web widget. Maybe it has to do with the new mouse-transparency functionality or my input mode?

                            Comment


                              #29
                              Sorted out the last issue, that was my mistake

                              What I currently struggle with is the ue4 function on my android phone, I've applied the JS required to make it work however it only works on websites on the internet / local webservers where there is a proper URL, it does not work for me when loading the HTML directly or loading a local file.

                              I've tried quite a lot, but it only returns undefined and no communication to UE4 is being picked up, I've applied events to both the interface and the URLchange and none of them trigger.

                              It's using Android Webview 52
                              Attached Files
                              Last edited by Tumenor; 09-11-2019, 07:36 AM.

                              Comment


                                #30
                                Originally posted by DsyD View Post
                                I seem to be having trouble getting range sliders to work properly. When mousing down on the slider "handle" and dragging, it may move for a short distance but then shortly stops responding. It seems like it stops processing if the mouse cursor moves off of the handle. This is on 4.22.3 editor on Windows 10, with the player controller input mode set to Input Mode Game and UI, with mouse transparency enabled on the web widget. Maybe it has to do with the new mouse-transparency functionality or my input mode?
                                You are correct that the range sliders are fidgety as I have experienced this issue with the example project as well. It was happening well before the transparency update and the new dynamic mouse transparency didn't seem to change the way it bugs out. The entire browser widget is just very buggy when it comes to drag-n-drop operations so there hasn't been a way to address this issue directly in the engine.

                                However in terms of a workaround there are a few things you can do. The easiest is to just not use range sliders or other weird input elements at all and just use JavaScript for everything. We actually use the jQuery UI for our form elements including range sliders and it removed a lot of hassle. Otherwise another potential workaround if you absolutely need to support native input elements could be to implement some kind of mouse position tracker in JavaScript which would manually remove focus from the slider if the mouse moves too far away. This could be achieved by manually triggering the mouseup event on the input element. But with that being said the jQuery UI is probably your best option with the fewest headaches.

                                Comment

                                Working...
                                X