Question about the lip-sync for the new expressive avatar — Oculus
Welcome to the Oculus Developer Forums!

Your participation on the forum is subject to the Oculus Code of Conduct.

In general, please be respectful and kind. If you violate the Oculus Code of Conduct, your access to the developer forums may be revoked at the discretion of Oculus staff.

Question about the lip-sync for the new expressive avatar

Hi,
I have followed the developer guide for the new expressive avatar update and every thing works well except the lip-sync feature.
In this document, it only mentioned that the expressive avatar utilized the OVRLipsync to enable realistic lip-syncing. However, it didn't mention how we actually make it work for the expressive avatar (e.g. what prefab or scripts that we should looked at and attach on the avatar)

Any help or suggestion would be really appreciated.

Best Regards,
Eric Wang


Tagged:

Best Answer

Answers

  • stevehinanstevehinan Posts: 115
    Art3mis
    As far as I can tell, all you need to do is set two bools to true in the Inspector for the Avatar Component itself: "Enable Expressive" and "Can Own Microphone". If you look at a local driver-driven third person instance with these settings, you should see it working.
    Thanks @cloud_canvas, is there another microphone input step required besides toggling the "Can own microphone"?  I can't get the avatar mouth to move yet and cant tell if there is any voice audio input from the headset mic.  I have allowed permissions on Android.
  • cloud_canvascloud_canvas Posts: 67
    Hiro Protagonist
    Are you including OVRLipSync from the Utilities too? The new version of the OvrAvatar Component makes use of it so you need to have it in your project.
    Samsung Galaxy S8 (Snapdragon 835), Gear VR (2017), Oculus Go (64GB), Unity 2018.3.14f1
  • stevehinanstevehinan Posts: 115
    Art3mis
    Are you including OVRLipSync from the Utilities too? The new version of the OvrAvatar Component makes use of it so you need to have it in your project.
    Yes, I do have OVRLipSync in my project, is there a component I need to add to the localavatar game object?
  • ericwang0701ericwang0701 Posts: 3
    NerveGear
    @cloud_canvas and @stevehinan Thank you for your reply. I followed the @cloud_canvas suggestion and finally make the lip-sync works.

    However, I did found a weird thing that if I run my scene in the editor mode, and if I select LocalAvatar or RemoteAvatar in the scene hierarchy to check some info, the lip-sync just stopped working. I guess I just need to prevent doing that at the moment.  

  • ericwang0701ericwang0701 Posts: 3
    NerveGear
    @stevehinan one easy way to test whether the lip-sync is working. You can use the RemoteLoopback sample scene in the Avatar SDK (i.e. Avatar/Sample folder). Hope you find a way to fix your issue.
  • stevehinanstevehinan Posts: 115
    Art3mis
    @ericwang0701 @cloud_canvas
    Solved.  Thank you both!
    In a nutshell, the OVRAvatar script requires the OvrAvatarLocalDriver component to be attached for voice to work.
    I had created a modified OvrAvatarLocalDriver script with a different name locking the position of the avatar.
    Therefore I just added the required OvrAvatarLocalDriver script back onto the OVRAvatar object as a duplicate component and it works.

    From OVRAvatar script:

    if (GetComponent<OvrAvatarLocalDriver>() != null)
            {
                // Use mic.
                lipsyncContext.audioLoopback = false;
                if (CanOwnMicrophone && IsValidMic())
                {
                    micInput = MouthAnchor.gameObject.AddComponent<OVRLipSyncMicInput>();
                    micInput.enableMicSelectionGUI = false;
                    micInput.MicFrequency = 44100;
                    micInput.micControl = OVRLipSyncMicInput.micActivation.ConstantSpeak;
                }
            }


    Steve Hinan - METAL MULTIBALL

  • legolasshegolaslegolasshegolas Posts: 10
    NerveGear
    I've followed all the steps in this thread, have the entire Oculus Utilities folder imported in my project (includes LipSync and Avatar) but the lipsync still doesn't seem to be working. I'm using the OculusGo and have my Avatar in 3rd person mode. The app asks me for permission to access microphone at launch as well but there's no lip sync. Just the Avatar's hands are moving. The head isn't even turning based on movement of the headset. 

    Please help?
  • stevehinanstevehinan Posts: 115
    Art3mis
    @legolasshegolas
    Do you still need help?  Have you got the head to turn as you would expect?
  • legolasshegolaslegolasshegolas Posts: 10
    NerveGear
    @stevehinan
    Yes, please. The head doesn't turn either. The hands are moving with the controller.
  • legolasshegolaslegolasshegolas Posts: 10
    NerveGear
    Okay, I have avatar working. Head and hands have movement.
    For the head to move, I had to set Texture Compression to ASTC in Build Settings. But one problem that has risen from this is that the avatar disappears on it's own in a few seconds after the app is launched.

    I still don't have the LipSync working. Don't know how to get the LipSync library working with the Avatar lib. Resources about how to get that working would be appreciated!

  • stevehinanstevehinan Posts: 115
    Art3mis
    @legolasshegolas
    I have encountered a lipsync issue that might be similar:
    When I freshly install my app on Oculus Go and grant microphone permissions via popup, I don't see the lipsync working.  But when I close the app and start it up again the lipsync works as expected.  
  • cloud_canvascloud_canvas Posts: 67
    Hiro Protagonist
    My app has a lot of issues associated with that as well. Unity only recently added support for Android runtime permission requests, and their implementation has a few bugs. In the case of RECORD_AUDIO (used for microphone access), you can't stuff the request up top before the app launches anymore where you used to be able to.
    You may want to pin your use of microphone functionality around a coroutine or an async Task that uses WaitUntil() that is waiting for the user to grant microphone permission to do whatever initialization behavior is necessary. (I guess in LipSync?)
    I hadn't been having the same issue as you guys, but it's probably because my app is using Vivox which is asking for the microphone permission well before an OvrAvatar would have spawned and thus fired up the LipSync .aars to tap into the mic.
    Samsung Galaxy S8 (Snapdragon 835), Gear VR (2017), Oculus Go (64GB), Unity 2018.3.14f1
  • legolasshegolaslegolasshegolas Posts: 10
    NerveGear
    So I'm trying to follow the instructions here to try to get the lipsync working: https://developer.oculus.com/documentation/audiosdk/latest/concepts/audio-ovrlipsync-sample-details/

    I can't figure this part out:
    "OVRLipSyncContext must be added to each GameObject which has the morph or texture target that you want to control.

    OVRLipSyncContextMorphTarget and OVRLipSyncContextTextureFlip are the scripts that bridge the viseme output from OVRLipSyncContext."

    Do avatars even have facial blendshapes? If yes, how do I access them for the OVRLipSyncContextMorphTarget script?

  • legolasshegolaslegolasshegolas Posts: 10
    NerveGear
    Also, why does the LocalAvatar gameobject disappear for me in a couple of seconds when I have Texture Compression set to ASTC in Build Settings
  • stevehinanstevehinan Posts: 115
    Art3mis
    @legolasshegolas
    If you are using a recent Unity Oculus Ingegration you should be able to do all the lipsync within the OVRAvatar script via the inspector.  Check enable Expressive, Can own microphone, Enable mouth vertex animation.
    I believe that's all you need to do, as long as you have the Oculus Package imported into your project including the LipSync folder.  Here is a screengrab of my OVRAvatar settings in Unity, note the AudioSource. 

  • legolasshegolaslegolasshegolas Posts: 10
    NerveGear
    @stevehinan Thank you for the reply!
    Okay so what was causing the trouble was the OculusGo's microphone not working... >.> Restarted the headset and lipsync is working. 

    Still, the avatar disappears after a few seconds. What causes this is Texture Compression set to ASTC in Build Settings. Setting it to anything else makes the avatar's head to lose all motion. Doesn't disappear though.

    Also, how can I get the avatar lipsync to work with an audio file rather than live input from the microphone?
  • stevehinanstevehinan Posts: 115
    Art3mis
    @legolasshegolas
    To troubleshoot the avatar disappearing I would create a new project and use the Oculus provided prefabs to see if that works as expected, if yes - see what is different with your project, if no - maybe a project setting?

    LipSync from an audio file, I have not tried this before.  Please share if you find a solution.
  • GigaSoraGigaSora Posts: 7 Oculus Start Member
    For anyone still having trouble after all that, here's what I've found after a little ablation testing. For Lip-sync to work out of box with the avatar system, you must:
    - have enable expressive selected
    - can own microphone selected
    - game view must be selected if in editor, no scene view or even clicking away
    - your local avatar object must have the ovr local avatar driver component on it
    - It does not appear you can have multiple local avatar drivers in the scene, though there was some ambiguity around that
  • legolasshegolaslegolasshegolas Posts: 10
    NerveGear
    @stevehinan Thank you for the reply!
    Okay so what was causing the trouble was the OculusGo's microphone not working... >.> Restarted the headset and lipsync is working. 

    Still, the avatar disappears after a few seconds. What causes this is Texture Compression set to ASTC in Build Settings. Setting it to anything else makes the avatar's head to lose all motion. Doesn't disappear though.

    Also, how can I get the avatar lipsync to work with an audio file rather than live input from the microphone?
    @GigaSora
    Do you have any idea about these issues I'm having? Any help would be appreciated!
Sign In or Register to comment.