Unity SG_HandPose reference

This page shows you how to access and use a SG_HandPose from the SenseGlove Unity Plugin. The SG_HandPose contains the pose data of a hand, taken from the native SenseGlove API and converted into Unity’s Coordinate System. This is different from the SGCore.HandPose class included in our native C# and C++ APIs, which uses the internal SenseGlove coordinate system. If you’re using the native APIs, you should go here.

The SG_HandPose class contains all data one might need to represent a hand in 3D space.

Accessing the SG_HandPose

To access a SG_HandPose, you’ll need a reference to a SG_HapticGlove script or, for a more generic approach, to a IHandPoseProvider interface. Like all SenseGlove scripts, these exist in the SG namespace. Note that at the moment, it’s not possible to access a SG_HandPose without scripting. Because the SG_HapticGlove implements the IHandPoseProvider interface, you can access the HandPose from either of them, using the following function:

/// <summary> Retrieve the latest SG_HandPose from this provider. </summary>
/// <param name="handPose"> When returning true, this handPose will contain the latest hand pose data from this device. </param>
/// <param name="forcedUpdate"> If true, we force a new update even through we already retrieved a pose this frame. </param>
/// <returns> Returns true when a handPose could be created from the device. Returns false if this method fails for a multitude of reasons (device is turned off, disconnected, etc). </returns>
bool GetHandPose(out SG_HandPose handPose, bool forcedUpdate = false);

You can call this function at any time. If the function returns true, a HandPose has been successfully retrieved. Below is an example of a script that calls it evey update and reports the HandPose.

using UnityEngine;
using SG; // Access to SenseGlove classes.

public class Ex_GetHandPose : MonoBehaviour
{
    public SG_HapticGlove hapticGlove;

    // Update is called once per frame
    void Update ()
    {
        SG_HandPose latestPose;
        if (hapticGlove.GetHandPose(out latestPose))
        {
            Debug.Log("Retrieved " + latestPose.ToString());
        }
        else
        {
            Debug.Log("Could not retrieve a handPose from " + hapticGlove.name + ". It's probably not (yet) connected.");
        }
    }
}

You can do the same using a IHandPoseProvider interface, although Unity does not allow us to link Interfaces through the inspector. Instead, we retrieve it off a GameObject during startup:

using UnityEngine;
using SG; // Access to SenseGlove classes.

public class Ex_GetHandPose : MonoBehaviour
{
    public GameObject handPoseSource;
    public IHandPoseProvider handPoseProvider;

    void Start()
    {
        handPoseProvider = handPoseSource.GetComponent<IHandPoseProvider>();
    }

    // Update is called once per frame
    void Update ()
    {
        SG_HandPose latestPose;
        if (handPoseProvider.GetHandPose(out latestPose))
        {
            Debug.Log("Retrieved " + latestPose.ToString());
        }
        else
        {
            Debug.Log("Could not retrieve a handPose from " + handPoseSource.name + ". It's probably not (yet) connected.");
        }
    }
}

Now that we can access a SG_HandPose, let’s look at the data contained within, and how we can use it.

Using a SG_HandPose

The SG_HandPose contains several different variables that can be used to represent a pose of the hand in 3D space. This guide assumes you are familliar with the anatomical notation of bones, joints and movements of the hand. If you aren’t, there’s an excellent reseach paper on the topic here.

Coordinate System

When the wrist rotation is 0, 0, 0, the following is true:

• The X-Axis runs parallel with the stretched fingers, running roughly from the wrist joint to the middle finger MCP joint.

• The Y-Axis points ‘up’ from the hand palm. The positive direction is from the palm of the hand to the back of the hand.

• The Z-Axis runs along the MCP joints. For right hands, the positive direction is from pinky MCP to index finger MCP. For left hands, the positive direction is from index MCP to the index finger MCP.

Note

Our left- and right hands share the same ‘forward’ and ‘upward’ directions. The difference in positive y-direction between left- and right hands means that finger flexion also shares the same positive / negative direction between the hands.

Using Arrays

Several of the variables in the SG_HandPose are contained within arrays T[] or nested arrays T[][].

In both cases, the arrays will have a length of 5, and the first index will always be used to indicate the finger, sorted from thumb [0] to pinky [4]. Imagine accessing these as [finger].

For nested arrays, the second index will always indicate the finger joint, sorted from most proximal (closest to the wrist) to distal (furthest from the wrist). Imagine accessing these as [finger][joint]. These nested arrays are of length 5x4, with the exception of the jointAngles variable, which is of length 5x3 - as these do not include the fingertips. In the thumb, the joint index is used to access the { Thumb CMC [0], Thumb MCP [1], Thumb IP [2], Thumb Tip [3] } joints. For the fingers, the joint index is used to access the {Finger MCP [0], Finger PIP [1], Finger DIP [2], Finger Tip [3]} joints.

Note

We use nested arrays because it’s easier to iterate over each finger. No need to calculate an index based on which finger you’re on. If we were to add additional joints to the thumb or pinky, the other fingers are also not affected.

RightHanded

/// <summary> Whether or not this pose was made for a right or left hand. </summary>
public bool rightHanded;

As the description implies, the rightHanded parameter lets us know if this SG_HandPose was created for a right hand (true) or left hand (false).

Usually, you’ll already know which hand you’re retrieveing data from, or use the TracksRightHand() function of the IHandPoseProvider. It is still useful to have when you’re loading a SG_HandPose from disk, or as a sanity check.

WristPosition and WristRotation

/// <summary> The position of the wrist in world space. </summary>
public Vector3 wristPosition;
/// <summary> The rotation of the wrist in world space. </summary>
public Quaternion wristRotation;

These variables represent the location of the wrist in world space. The wrist position is given in meters [m]. You can apply the rotation and position to a GameObject to place it at the wrist location.

When accessing a SG_HandPose from a SG_HapticGlove, these are calculated based on your tracker / controller location, with additional hard-coded (or custom) offsets.

NormalizedFlexions

/// <summary> The total flexion of each finger, normalized to values between 0 (fingers fully extended) and 1 (fingers fully flexed).
/// The index [0..4] determines the finger (thumb..pinky). </summary>
public float[] normalizedFlexion;

The normalizedFlexion variable is the simplest representation of joint movements within the API. It consists of five values which represent the level of flexion (a.k.a. finger bending) in the range of [0…1]. This array is sorted by finger, from thumb [0] to pinky [4].

By summing all of the flexion angles of the finger, and dividing these by the maximum total flexion achievable within the finger’s natural limits, we get the normalizedFlexion. A value of 0 represents a fully straightened finger, parallel to the palm, while a value of 1 represents a fully flexed (bent) finger. A value in between, 0.25 for example, represents a finger that is roughly 25% bent.

It can be used for simple gesture or intent detection, and even has its own separate access function in the IHandPoseProvider interface:

/// <summary> Retrieve Normalized Flexions of this provider. </summary>
/// <param name="flexions"> An array that will be of length 5, containing flexion values normalized between 0...1. </param>
/// <returns> True if flexions is properly retrieved by this provider. </returns>
bool GetNormalizedFlexion(out float[] flexions);

JointAngles

/// <summary> The angles of each joint, in degrees, where flexing the finger creates a negative z-rotation.
/// The first index [0..4] determines the finger (thumb..pinky), while the second [0..2]  determines joint (CMC, MCP, IP for thumb. MCP, PIP, DIP for fingers.) </summary>
public Vector3[][] jointAngles;

The jointAngles variable contains the Euler angles of each joint of each finger, in degrees, relative to the previous joint, contained in a Vector3. For the first joints of each finger, the angles are relative to the wrist reference frame. It’s size is 5x3.

• x represents the joint’s pronation / supination (twisting), in degrees, relative to the previous joint. It’s mainly relevant for the thumb’s CMC joint (jointAngles[0][0].x).

• y represents the joint’s abduction / adduction (sideways motion) in degrees, relative to the previous joint. It’s mainly relevant for the first joint of each finger (jointAngles[finger][0]).

• z represents the joint’s flexion / extension (finger bending) in degrees, relative to the previous joint. For both hands, extension is positive and flexion is negative.

The values of these angles are limited to “normal” human ranges:

Relevant jointAngle Ranges

Movement Name	Joint Index	Range Left [°]	Range Right [°]
Thumb CMC Twist	jointAngles[0][0].x	-20 … 20	-20 … 20
Thumb CMC Abduction	jointAngles[0][0].y	-10 … 60	-60 … 10
Thumb CMC Flexion	jointAngles[0][0].z	-35 … 10	-35 … 10
Thumb MCP Flexion	jointAngles[0][1].z	-50 … 0	-50 … 0
Thumb IP Flexion	jointAngles[0][2].z	-90 … 10	-90 … 10
Finger MCP Abduction	jointAngles[finger][0].y	-20 … 20	-20 … 20
Finger MCP Flexion	jointAngles[finger][0].z	-90 … 30	-90 … 30
Finger PIP Flexion	jointAngles[finger][1].z	-100 … 0	-100 … 0
Finger DIP Flexion	jointAngles[finger][2].z	-90 … 5	-90 … 5

You can turn these euler angles into quaternions using Unity’s Quaternion.Euler(Vector3) function. This is the simplest way of retrieving relative Quaternion rotations if you’re working with localRotation as opposed to (world) rotation.

JointRotations

/// <summary> The quaternion rotation of each joint, relative to a Wrist Transform: JointRotation * WristRotation = 3D Rotation.
/// The first index [0..4] determines the finger (thumb..pinky), while the second [0..2]  determines joint (CMC, MCP, IP for thumb. MCP, PIP, DIP for fingers.) </summary>
public Quaternion[][] jointRotations;

The jointRotations are the Quaternion rotations of each finger’s joints, all of which are relative to the wrist. It’s size is 5x4. They are calculated this way within the native C# API, and allow us to apply the quaternion rotation directly to each joint, using something along these lines:

for (int f=0; f<5; f++) //finger (f), [0..4].
{
    for (int j=0; j<4; j++) //Joint Index; proximal to distal, where 4 = fingerTip.
    {
        jointTransforms[f][j].rotation = handPose.wristRotation * handPose.jointRotations[f][j]; //assuming jointTransforms is a (nested) array containing the valid transforms.
    }
}

This is similar to how we apply the rotations in the Unity Prefab models. For a deeper understanding, we suggest looking at the SG_HandAnimator class.

If these quaternions are identity quaternions (Quaternion.identity or [0,0,0,1]); all fingers are stretched and parallel to the hand palm. The thumb is a special case: When the CMC joint rotation is an identity quaternion (all angles 0), the thumb is also stretched parallel to the fingers, and it has an identical flexion direction as a finger. This is because a “starting position” of the thumb is incorporated in Quaternion at jointRotations[0][0]. This starting rotation is a Quaternion made form Euler angles [-90,0,0] (left hands) and [90,0,0] (right hands). We encourage you to look at the starting pose of the Prefab Hand Models inside the Unity Plugin to see exactly what is meant by this.

JointPositions

/// <summary> The position of each joint, in meters, relative to a Wrist Transform: (JointPosition * WristRotation) + WristPosition = 3D Position.
/// The first index [0..4] determines the finger (thumb..pinky), while the second [0..2]  determines joint (CMC, MCP, IP for thumb. MCP, PIP, DIP for fingers.) </summary>
public Vector3[][] jointPositions;

The jointPositions variable is a nested array of Vector3’s, which represent the position of each finger’s joint, all of which are relative to the wrist. It’s size is 5x4.

Unless your 3D hand model does not use parenting for its bone structure, you’ll likely not need to set the position of any joint(s). In most cases, Unity will calculate the new position(s) based on the joint’s rotation(s).

If you do wish to apply the positions to a hand model, or are simply looking to draw a wireframe model of the hand, you can calculate the joint position in world space using the following method:

for (int f=0; f<5; f++) //finger (f), [0..4].
{
    for (int j=0; j<4; j++) //Joint Index; proximal to distal, where 4 = fingerTip.
    {
        jointTransforms[f][j].position =  handPose.wristPosition + (handPose.wristRotation * handPose.jointPositions[f][j]); //assuming jointTransforms is a (nested) array containing the valid transforms.
    }
}

Serialization

When you retrieve your SG_HandPose, you can convert it into a string notation using the Serialize() function on the instance. This will condense all relevant data into a single line of text, that can be stored on disk or hard-coded, and unpacked later.

The static Deserialize() function can be used to convert a string notation back into a HandPose class.

/// <summary> Create a string representation of this HandPose to store on disk. It can later be deserialized. </summary>
/// <returns></returns>
public string Serialize()


/// <summary> Unpack a HandPose back into its Class representation. </summary>
/// <param name="serializedString"></param>
/// <returns></returns>
public static SG_HandPose Deserialize(string serializedString)

// Example Implementation
public void TestSerialization( SG_HandPose pose )
{
    string serializedPose = pose.Serialize();
    //I can now store serializedPose in a text file, for example

    SG_HandPose loadedPose = SG_HandPose.Deserialize(serializedPose);
    //LoadedPose should be identical to the original

    bool equalPoses = loadedPose.Equals(pose);
}