cancel
Showing results for 
Search instead for 
Did you mean: 

On-device Speech Recognition on the Quest with Unity

Anonymous
Not applicable
Hi, I am working in Unity, and I have been trying to implement on-device speech recognition on the Oculus Quest. Knowing that the Quest uses Android, I tried using Android's speech API's through a plugin I found on the Unity asset store. The plugin gives me speech recognition on my Android Phone but not on the Oculus Quest. On the Quest I get an error saying that speech is not supported. Looking through the forums, I learned that the Oculus Go is a "bare bones" version of Android that doesn't have native Speech Recognition. I've been searching for other on-device speech recognition solutions, but so far it seems like most speech recognition solutions require the internet.
Can anyone recommend an on-device speech recognition solution that is compatible with the Quest, if one exists? Or will I have to go to an online-based solution?
11 REPLIES 11

Anonymous
Not applicable
Bump

ethanmh
Honored Guest
I'm also very interested in finding an on-device speech recognition solution that is compatible with the Quest. Following.

pdehaanST
Honored Guest
I can verify that android builds of Pocketsphinx work on the quest. IME with Unity the Swig/JNI wrappers complicate things, you can just drop in the appropriate shared libraries and use C# interop (DllImport and such) to call into libraries straight from C# code.

Still have the usual mobile device limitations (e.g. don't use massive vocabularies), but works fine otherwise.

ethanmh
Honored Guest
@pdehaanST How did you declare the functions to call? Did you have to use swift? Do you have an example of initializing I could see?

pdehaanST
Honored Guest
No swift involved for android platforms. Most of the work is handled by C# attributes from System.Runtime.InteropServices, for example:
using System.RuntimeInteropServices;

public static class SphinxAPI {
    [DllImport("pocketsphinx_jni")]
    public static extern IntPtr ps_args();

    [DllImport("pocketsphinx_jni")]
    public static extern IntPtr cmd_ln_init(IntPtr inout_cmdln, IntPtr defn, Int32 struct, IntPtr _nl);
}

...declares functions that I can call from my C# code that are mapped to plain old C functions from the pocketsphinx_jni shared native library. So somewhere in a setup function I've written, I've got a line like:
var config = SphinxAPI.cmd_ln_init(IntPtr.Zero, SphinxAPI.ps_args(), 1, IntPtr.Zero);

IntPtr is basically just a C# wrapper around pointers for the C code. In the pocketsphinx library cmd_ln_init returns a cmd_ln_t* (see https://cmusphinx.github.io/doc/sphinxbase/cmd__ln_8h.html and https://cmusphinx.github.io/doc/sphinxbase/cmd__ln_8h.html as potential starting points for API docs), but my C# code doesn't know about that, I just need to store the pointer to pass into other functions.

I've got a slightly more fleshed out example of a Unity project at https://github.com/pdehn/UnitySphinx, but that's currently Windows-only. You'll need a version of the libraries compiled for android to use with the quest (a quick solution is to just grab the .aar from the cmusphinx android demo repository at https://github.com/cmusphinx/pocketsphinx-android-demo/blob/master/aars/pocketsphinx-android-5prealp...). You'd likely want to overhaul the code from my demo project as well. Of note, it runs all decoding in the Unity main thread (can lead to frame drops which is bad for VR) and you'd need to unpack all files that pocketsphinx reads (acoustic models, dictionary, language model, etc.) onto the filesystem, since Unity just packs everything into a .apk that sphinx wouldn't know how to read from.

In theory you could also use the Java bindings generated using SWIG, but I've not touched that and can't really help there.

Anonymous
Not applicable
@pdehaanST Thanks for the recommendation! I was able to get a working solution of PocketSphinx running on an Android phone using a project someone else made here. I won't be able to test it on the Quest until I get access to it next week. I'll post how it goes.

Another solution recommended to me was to use TensorFlow Lite, but I haven't looked into it yet.

checkbox
Honored Guest
Ahzee, can you post updates on how far you got? Really interested in this ?

LeonKleinBretel
Explorer
Hi guys, any update on speech recognition? on-device speech recognition on the quest would be really useful.

ognjen_todic
Honored Guest
My company (Keen Research) develops SDKs for on-device speech recognition for Android and iOS; we also offer Unity plugin. We license SDKs on commercial bases and offer a trial version of the SDK (works for 15min at the time) for evaluation purposes.

Forum doesn't allow me to post links but you can use your favorite search engine and search for KeenASR.

/Ogi