cancel
Showing results for 
Search instead for 
Did you mean: 

Speech Recognition in GEAR VR. Tips, Tools & Tricks?

SilkWebware
Explorer
Hi,

Has anyone tried using speech recognition in a GEAR VR app?
Is it even possible with the CPU constraints? My graphics are such that I can afford to drop the timeWarp vsync rate to support 30Hz if necessary.

I'm looking for tips, tricks and tools.

Thanks in advance.
Nick
Gear VR development blog at https://gearvrdevelopment.wordpress.com/ The first article is to help new comers set up their development environment using Unity 5.3 https://www.linkedin.com/in/nickharrow Nick Harrow (I'm always open to new opportunites)
19 REPLIES 19

delphinius81
Rising Star
For SR, pocketsphinx (android version of cmusphinx) is a good choice. It's a pain to get built and integrated with Unity though - someone on a project a worked on at a previous job spent weeks getting it working, but he was Unity novice. However, if you can make it through the build process for pocketsphinx android and know how to create a unity plugin, it's rather powerful. Plus, it will run locally on device, so you don't have to worry about network latency. Recognition times were pretty quick, even on mobile, but that is largely because you provide the (limited) grammar for it to recognize.

Honestly, if you managed to get it built and working as a Unity plugin, I'd love to use it too! 🙂

play_edu
Explorer

leftler said:

I know the Samsung Internet Beta uses it as an alternative for the keyboard input.



Also in the Oculus Sample Framework for Unity 5 there is a keyboard example, I have not had the time to look at it yet but it may contain a voice recognition part too (again, I have no idea if it does, this is just speculation).

in Oculus Sample Framework for Unity 5 there is only keyboard demo, not voice API demo.
also, I try many other demos but not found any Speech Recognition demo. So I mad android native voice API Plugin but  Plugin work with gear VR not with oculus go.  any buddy has the solution for Oculus Go. If need then I can ready to parches the plugin. sorry for my bad English.

play_edu
Explorer

leftler said:


play_edu
Explorer
...

rdmitchell1997
Expert Protege
I would definitely take a look at IBM Watson and what that as a service has to offer. it is incredibly versatile and allows speech recognition and also uses machine learning to create a natural speech system, for example, if you programmed it to listen for the word 'Hi' the machine learning would also know that the word 'Hiya' and 'Hello' had the same meaning.

it is quite hard to explain but the IBM Watson Sandbox demo explains it best. hope this helps!
Sincerely,

Robert Mitchell

Email: rdmitchell@blueyonder.co.uk
Website: http://vr.port.ac.uk/

play_edu
Explorer
HI,
thanks for replay.
IBM Watson or google voice Sevice provide nice API but its only work with internet not offline. Android device speech engine works great and also offline and free but its work on Android devices(gear vr) not on oculus go. 

Best regards,
Play_edu

SmartDigitalNet
Explorer
I wonder if part of the solution is buried in the Audio SDK in the Oculus Lipsync Code. Seem there is some code listening for 15 visemes. Maybe this function can be rewritten to do other things when it hears the 15 sounds and or maybe even change what it is listening for. https://developer.oculus.com/documentation/audiosdk/latest/concepts/audio-ovrlipsync-sample-details/

Other Oculus Lipsync Scripts

OVRLipSyncMicInput is for use with a GameObject which has an AudioSource attached to it. It takes input from any attached microphone and pipes it through the AudioSource.

We recommend looking at the other scripts and prefabs included with this integration. They will provide more insight as to what is possible with Oculus Lipsync. We include, for example, some helper scripts to facilitate easy on-screen (in VR) debugging.

motorsep
Rising Star
@SmartDigitalNetworks

I don't believe it's supported in UE4 and it's only to do lipsync. It's not going to recognize voice.

Nishantbtps
Honored Guest
Can anyone Give a solution for using Android Speech engine in Gear Vr?? 

legolasshegolas
Honored Guest
Hey! I have lip sync working with existing avatars on the Oculus Go in Unity but I can't seem to switch the audio source from microphone to audio file. I'm not even using the OVRLipSyncMicInput script. I'm using the OVRLipSyncContext script but source is still the microphone.

Also, the avatar disappears after a few seconds of the app booting. What causes this is Texture Compression set to ASTC in Build Settings. Setting it to anything else makes the avatar's head to lose all motion. Doesn't disappear though.