Thank you! Total hero!!
I realise this reply comes a bit late for most participants of the topic, but I figured it out thanks to roto’s excellent post and then saw some confused users, so for anyone else in the future Googling this…
Here’s my full repro for you:
-
In Maya, create a sphere at (0,0,0). Create two joints. Parent one joint to the other, and ensure correct joint orients (Attribute Editor > Joint > Joint Orient – I did (-90, -90, 0) so Z is up and X is forwards). Set transforms to (0,0,0).
-
Rotate the sphere so one of its poles faces down the Z axis. Freeze transforms.
-
Select both joints and the mesh, and Bind Skin the mesh to both joints.
-
Group all four objects together (CTRL/CMD+G). This may break the hierarchy of the joints. Reparent if necessary.
-
In Rigging Mode, go into Skin > Paint Skin Weights.
-
In the Tool Settings (top-right, hammer with the three dots coming out of it), select the parent joint, and hit “Select Geometry”. Flood all influences to 1.0.
-
Select the faces and/or vertices around the forward-facing pole. Select one or two more rings as per your liking. This will be your pupil. Select the child joint, and flood influences to 1.0.
-
Select the next ring of verts and use Paint Operation: Smooth. Smooth three steps. Next ring, smooth two steps. One more ring, smooth one step. We’re doing this to avoid shearing of the pupil geometry when we scale its joint up and down for cat eye and/or pupil dilation.
-
For debug purposes to visualise where the pupil is in Unreal Engine, go into the UV Editor for the mesh and create a new UV set: UV Sets > Create Empty UV Set.
-
Go back to the first UV set, and select the faces that match with the pole you used for the pupil. In my case, it was the top three rows of faces. Cut these from the first UV set.
-
In the second UV set, paste them. In my case, they ended up all overlapping. For production, you’d want to clean this up so you can actually texture your pupil, but since I’ll be using flat colours in this tech demo, I don’t care.
-
Time to export! File > Export All… → choose FBX format, and appropriate settings (e.g. Geometry > Smoothing Groups YES; Cameras NO; Lights NO; Audio NO; Axis Conversion you can leave on Y up (UE will fix this in Import)).
-
In Unreal, Import your FBX as Skeletal Mesh.
-
Right-click the eyeball Skeletal Mesh and Create > Control Rig.
-
Right-click the parent joint and create a control, then right-click the child joint and create a control. Unparent both of these, then make the control for the parent joint the parent of the control for the child joint. You never want controls in the same space as the bones, to avoid double transforms!!
-
Set up the controls so they drive the appropriate bones; drag a control into the graph and click “Get control”, then drag the bone you want it to drive into the graph and click “Set Bone”. Link them up from the Forwards Solve and attach the Transform pin from a Get Control node to its Set Bone node.
-
Make the controls nice to use and see; e.g. use the Ring_Thin shape, and move them a little bit in front of the eye. I will explain why mine are a bit oblong in a moment.
-
Make a Blueprint of type Character. Import your skeletal mesh of choice, then create a new child on the Mesh (CharacterMesh0) component of type Skeletal Mesh. Choose the eyeball you just imported. Put it in place, and scale it around a little so that it becomes clearly not-spherical. You want to be sure this effect is working as intended.
-
If you now try to rotate this non-spherical eye, you’d run into problems as expected: the weird shape of the eye starts clipping through the mesh in all sorts of ways. This is the problem we’re solving! Keep going.
-
Only one more step, really! Drag your Blueprint into the Level Editor, then open Sequencer and add your Character to the Sequencer. Twiddle down the controls for the eye Skeletal Mesh and find your parent bone control. Rotate it around… magic!!
(You can see here that my controls are round instead of oblong. I inverse-scaled them in the control rig so they’d turn out round in Sequencer after being scaled in the Character Blueprint.)
As promised by our Pixar hero, we have now set up a system where, as far as the eye is concerned, it’s first deforming according to its spherical shape, i.e. rotating as a sphere would. And then Unreal goes “actually, you’re an oblong spheroid inside this character’s head”. The order of deformations is correct: we get a rotating eyeball that is non-spherical. BEAUTIFUL!!