Hey guys,
I could work ages on this, but at some point it is important to take the next steps, make something public and specialize - and I guess the time comes nearer.
So, here is my application called OvrRemote - my contribution to epic games awesome pixelstreaming.
The possibilities of the combined features are in my opinion nearly endless. I tried to generalize the featureset as good as possible, in order to be able to smooth transit to various application-types and target-systems on demand.
But from now on, I need to have more feedback, so that in the future, I can specialize in a more unique direction.
I would very much like to hear your opinion.
How do you think can this app go into the wild?
What is missing ?
Where do you see a good use-case ?
Or do you think, that it is (still) too early for (VR) remote collaboration? Probably a rhetorical question… or partially rhetorical. Anyway…
Because of the amount of information, I have to split it.
This part is focussing on core-features and includes pictures, and the the other part coming later will focus on a more detailed technical background.
Please let me know if you need a live presentation.
The application is not ready to publish yet - although most features are now implemented and all core features were tested.
**Background: **
About half year ago, I was very impressed by the application “Bigscreen” - which allows to watch movies with friends in a virtual theater enviroment. I immediately wanted to code something similar, but also improve the features a bit at the same time. I was not aware of the implications and am still excited. I do not think that developing in this field can be considered a risky investion any longer.
I knew a long time about the “presence” feeling in VR, but refused to translate this to a social VR exerience. I guess that, although I am a VR enthusiast since the early
days, I was still too biased for trying this out.
Now I am very sure that using VR is an important aspect of (future) communication. Of course, there still exist limits of how many information can be transfered digitally, but it certainly can already transport more useful information than the common existing 2d-collaboration tools and is far far more than “a toy”.
In a not so far future, the uncanney valley of social avatars will be completely negligible as well. It is in my opinion only a matter of two features still missing in current VR generation.
The first is a hardware one: Eye tracking.
The second is a software one: Using the eye-tracking cameras to track facial expressions.
With realtime facial expressions, the uncanney valley will eventually go away, so that we will get to the core of 100% useful VR meetings soon.
(If you do not include other perceptions like smell or touch and if you can live without per pixel raytracing for another year or so.)
Yes, it is certainly not perfect yet. But it is still much more useful than staring on a flat webcam-grid. Not all features are necessary for a worthy experience.
So. What is “the right amount of VR” which is currently useful? If you have tried out bigscreen with reasonable VR hardware or if you have seen oculus avatars in action - this is probably nearly self-explanatory.
Otherwise, the topic is already well understood. Facebook has for example done a lot of research in this field, as there exist plenty of other papers, too.
See for example:
https://www.wired.com/story/facebook…ec-avatars-vr/
https://distantjob.com/blog/virtual-…k-environment/
https://www.researchgate.net/publica…irtual_Reality
Now to OvrRemote:
**OvrRemote **uses the unreal engine to let players jump into a mutiplayer-session in VR, in 3D and via a modern webbrowser (In 3D).
Any player on any platform can see, hear and communicate with the players on all other platforms at any time.
Screens with various content can be shared and edited at the same time.
For using VR and for having the best experience in 3D you need to download an application beforehand.
Thanks to the unreal engine, most platforms and (sub-)systems are already supported or can be easily supported with overviewable efforts.
For people without a dedicated GPU, it is also possible to enter a meeting in 3D by using the browser by simply entering an URL.
The rendering then happens on server-side and uses an overview-camera (which is scalable for many players), or can also give a limited number of players full access to all features of the downloaded application - i.e. a webplayer can play like normal a local player.
The latter is limited by the server graphics hardware, which limits the amount of web-meeting-participants having the full experience.
But, OvrRemote is not meant to compete with Skype, MS Teams, or WebVR apps.
Instead, it can be in my opinion seen as a bridge between classical and newer approaches, while you are already able to use the advanced features of the unreal engine to provide better and more beautiful content for medium sized meetings.
Other features are as far as I know also unique in the field of remote collaboration tools (see more below in detail).
The clue is, that you can use any graphical output device to enter a room and always see the same kind of surroundings and the same avatars while also being independent of your machines capabilities. It even works on mobile devices. As a (small) advantage, using server based rendering also does not drain the battery too much.
It is independent of steam or other commercial online-subsystems, as it can make use of any webserver or webhosting services.
Means, it is supposed to work seamlessly for all kind of platforms and subsystems.
Core-features:
- Remote desktop sharing for an arbitray amount of monitors
- Remote desktop editing with permissions for an arbitray amount of monitors (this is afaik unique)
- VR Keyboard with configurable color cycling
- advanced 3d-“Ambilight” for any desktop content using lightfunctions and screencapture-components.
- voice chat
- text chat
- customized onlinesession-subsystem for webserver using C++/PHP/SQL/Javascript
- optimized controls and desktop-editing mode in normal 3D. The camera moves in front of the 3d-desktop on interaction, so that you only see the monitor - which is indistinguishable from working on the desktop outside of the engine
- generic oculus avatars with voicechat-lipsync
- generic simple avatars
- you can move freely or use teleportation. The player has collision.
- Interactable objects with animated selectionshader
- Transparent toolbars for every interactable object which always work the same way
- client authoritative movement for VR-meshes.
- using Steam VR OR Oculus VR
- Generic desktop-actors can be spawned, configured and saved as a preset.
- Desktops can show: a webbrowser, ingame streamer-cams, any physical monitor, any media, attached webcams - everything in optional side-by-side mode and with changable opacity.
- Every content-type can be streamed.
- shared monitors can be edited.
- editing-permissions for shared monitors are configurable per user.
- desktopaudio is grabbed seperately from ingame-sounds and voiceaudio in order to avoid voice-echos on client-side
- webplayer voices are attenuated and shared between host and other clients. Webplayers hear the same as everybody (except spatialization)
- configurable pixelstreaming quality (FPS, Max bitrate, capture-resolution, MinQP, configurable audio-capture devices)
- “Last-played” info always shows the last active window-title on client
- Hiqh quality pixelstreaming-audio (48khz, stereo, no compression)
- In-game configurable streamer “follow-cameras” which can be broadcasted
- Per object configurable mute/solo/volume-sliders
- Avatar selector using planar mirrors
- A virtual google assistant using oculus avatars answering questions continously without having to listen for a keyword - unfortunately google does not allow to use this commercially.
Also the assistant uses a deprecated API. I am not quite sure what to do with it, but it is awesome. Maybe it could be used somehow in other constellations.
Current limits:
- Network is not optimized for many players yet
- Using Webplayers is limited and not generalized yet, since I am no expert for WWW/WebRTC programming.
- OvrRemote uses a heavily modified pixelstreaming-plugin for sharing the desktop-content, but still needs TURN and STUN servers to work in all network constellations. AFAIK there exist no public TURN servers, so I am not sure how to cope with too strict firewalls without a commercial one. I am already automatically forwarding ports and add firewall exceptions when starting to host, but this will not be sufficient for too strict firewall-policies…
Not implemented yet, but planned:
-
Switching of VR/3D mode on the fly, which would be useful for meeting-leaders to be able to type on keyboard outside vr and be present in vr at same time without having to restart the session
-
record vr meetings or vr lessons with configureable in-game-cameras. These in-game-cameras could be made more and more intelligent - currently they can follow and keep the line of sight - so that they would show the point of interest content-based. Ideally, you would want to have a voice analyzer which makes the camera cut or move dependent on the current speaker, or switch between multiple variants to fulfill a certain “movie-content-type”. A “director AI”, so to speak You could create spectator views which could be interesting for web-viewers, who in turn could jump anytime into the discussion and participate.
Maybe these spectators could only participate, when they would fulfill some criteria beforehand, like having a good reputation in order to avoid trolling and to increase the discussion quality, and only if the adminstrator currently allows it - for feedback for example. Depends on the discussion type
-“Channel/Room- Tags”, which could be used to search for a specific keywords when you search for open servers
Where I see the use-cases:
- An app like Bigscreen for virtual movies (the first intention, but then it escalated)
- virtual interactive TV-channels
- a global open virtual discussion platform. Search for a specific keyword, and find all kind of people who are currently talking about it. Being able to jump into a discussion of interest virtually immediately is a goal. Needs additional webserver programming.
- teammeetings for medium-sized teams (already possible)
- mentor/education applications. The spectator-count is only limited by the upload bandwith. The game can make use of 32 game-participants (bandwidth is not optimized yet. The true maximum player amount is not clear yet. Viewing a spectator-stream is only limited by upload bandwidth)
- pair-programming (two programmers can share the screen with half transparency, so that you can view the colleague on the other side of the monitor. Since you can edit the others monitor from two sides, you can swap your (working-)place seamless horizontally while maintaining body presence to the other participant) (works)
- due to the missing haptics of the keyboard in VR, the currently more realistic scenario: any collaborative design-sharing usecases and many collaborative desktop-content editing-workflows (works)
- due to the possibility of placing and cloning an arbitrary amount of desktop-monitors in virtual space, you can create your virtual master environment for any kind of pressional setups like audio-production with virtual keyboards, synths and track-editors in any orientation and size in space which suits your current composition and save this as a preset, so that you can have the same production-environment on click again (works)
- post-production/movie-editing
- using touch for “painting on screen” instead of using a virtual mouse improves some 2d workflows (not implemented yet)
I can propose timelines for most use-cases to get this to a beta stage - and to a final stage if I know the requirements.
Of course, remote applications are a highly changing field - but the necessary technologies are now mostly founded - or are going in the right direction - so that calculated timelines are relatively save and sane.
Just leave me a PM please.
This was a long text, but still not long enough to list all possibilites :/.
Thank you for reading!
Cheers,
Carsten
