OpenXR Hand Tracking EXT

[koujaku]

So I've got code that uses the OpenXR hand tracking extension, and it works perfectly with HoloLens 2 and the Leap Motion layer. On Quest, I can see the extension, xrGetSystemProperties reports it as available, I have the manifest settings configured and the app does not ask me to switch to controllers, but XrHandJointLocationsEXT::isActive always reports false when calling xrLocateHandJointsEXT!

Any thoughts as to what I'm missing here? Is there some additional configuration I may have missed, or is this perhaps still buggy?

Engine is custom, StereoKit, so it's native code. I'm using the 2020-12-04 (OpenXR 1.0.12) release of the Oculus OpenXR Mobile SDK.

[johnkearney]

Hi @koujaku,

Let's try to track down the basics here. In order to get hand tracking to work with your app on Quest, you'll need two (and a half things) things:

1. The user on the device needs to opt-in enable it at the system level.

2. The app needs to explicitly ask for support for it in the manifest.

3. hands and controllers cannot be on together

Let's assume that both of these things (1 and 2) are done and your controllers are not active. Validate this by ensuring that hand tracking is working in home - and that you can start your application using hands.

Let's assume that's done, then the behavior here is surprising - and I cannot replicate it. Are you seeing the `locationFlags` update with the tracking state for the hands as they come into view of the cameras?

Thanks,
John

[koujaku]

Hey there! Just updated to the latest 1.0.13 drop today, so I'm peeking at this again.

1 should be good to go. Hands work great in the home environment, I can see my hands bringing up the guardian boundary inside my app, and can trigger the home gesture from within my app.

I believe 2 should also be good. I have these Oculus specific lines in my manifest file:

<uses-permission android:name="oculus.permission.HAND_TRACKING" />
<uses-feature android:name="android.hardware.vr.headtracking" android:required="false" android:version="1" />
<uses-feature android:name="oculus.software.handtracking"     android:required="false" />

I've tried the handtracking 'required' property as both true and false.

For 3, my controllers aren't active, or used by the system. They'll turn on if I pick them up and start pressing buttons, and my code will recognize that they're present. I haven't tried this with the batteries out or anything yet though 🙂

Location flags have been coming in as zero for me. I tried watching a couple different joints here, but nothing was coming through there either.

I'll see if I can pick out any interesting tidbits from the logs as well!

[koujaku]

I just snipped out any piece of the logs that mentions hands. Some of it looks relevant, but I'm not too clear on what it means! If there's anything in particular you'd like more detail about, let me know.

----------- xrCreateInstance [start] ----------
xrCreateInstance: Extension XR_KHR_opengl_es_enable enabled.
xrCreateInstance: Extension XR_EXT_hand_tracking enabled.
xrCreateInstance: Extension XR_KHR_android_create_instance enabled.
Registering LifecycleListener
...
SYSTEMBUTTONSPACE GKS freeze '0'  freezeAPI '0'  eattrigger '0'
SYSTEMBUTTONSPACE 'com.companyname.stereokittest_android' Freeze input '0' eat trigger '0'  on home button down
HANDTRACKING: YES! You are on the Hand Tracking Frequency GK 😄
HANDTRACKING sethandtrackingfrequency 0
HANDTRACKING handtrackingsmallfingeropeningpinchenabled 1
HANDTRACKING: YES! You are on the Hand Tracking Small Finger Opening Pinch GK 😄
InitializeInputHooks
InitializeInputHooks create
...
[SK diagnostic] Switched to input source: OpenXR Articulated
------------ xrCreateHandTrackerEXT -----------
------------ xrCreateHandTrackerEXT -----------
[SK info] Initialization successful
...
nativeHeadsetEvent(0)
handPoseLatency0 - Logger reset. Session ID: 8352975376531104382
pointerPoseLatency0 - Logger reset. Session ID: -1134678192484877506
handPoseLatency1 - Logger reset. Session ID: 246103885738099021
pointerPoseLatency1 - Logger reset. Session ID: 123073152194988773
gfrLatency - Logger reset. Session ID: 8714201424862031600
trackedRemoteLatency0 - Logger reset. Session ID: 3089447224890184107
trackedRemoteLatency1 - Logger reset. Session ID: 6111308168069116856
DEADRECKONING SetControllerTrackingExtendedDeadreckoning 0
...
ovrInputDeviceManager::Initialize
OSSDKTRACKING Setting ResultUseOSSDKTracking to 0
HANDTRACKING packagename is com.companyname.stereokittest_android
OSSDKTRACKING Setting ResultUseOSSDKTracking to 0
HANDTRACKING:  'com.companyname.stereokittest_android' Is hands or controller.
HANDTRACKING: feature flag set to 2
HANDTRACKING sethandmanifest 2
onClientConnect
Connecting to TrackingService
...
ovrDistortionMeshHeader::leftEye = up 48.000000 / down 50.000000 / left 49.000000 / right 45.000000
ovrDistortionMeshHeader::rightEye = up 48.000000 / down 50.000000 / left 45.000000 / right 49.000000
HANDTRACKING Pose filter profile changed to: 0
HANDTRACKING Pose filter profile changed to: 0
PhaseSync::PhaseSync PaddingLowerLimit 0.350000 StandardDeviations 2.000000 PaddingUpperLimit 0.750000 on instance 0
PhaseSync Instance created

[koujaku]

@johnkearney I guess I would also add, I've chatted with at least 3 other OpenXR devs that have run into blocking issues with hand tracking on Quest. I don't know for sure they're seeing exactly the same problem, but at least one of them sounded a lot like what I've been running into!

If it's working reliably on your end, perhaps there's an issue with the documentation? Maybe you could point out a code sample or publicly available repository that you know for sure works?

[koujaku · Accepted solution]

Ok, found the solution! The permission string listed in the docs doesn't seem to work for me on Quest.

Here's the broken string I was trying to use:

<uses-permission android:name="oculus.permission.HAND_TRACKING" />

And here's what did end up working for me:

<uses-permission android:name="com.oculus.permission.HAND_TRACKING" />

[johnkearney]

@koujaku - Thanks for the heads up on this! I've updated the only reference I could find to this string in the docs.