Description
This plugin uses voice recognition engine to generate lip-sync animation from SoundWave assets or PCM audio data. Animation is saved as curves in data assets and can be played in runtime together with audio. This approach allows to achieve well-looking lip-sync animation easily without subtitles.
Additional feature: recognize input from microphone (speech-to-text) in runtime.
Video #2: Lip-sync Curves to Anim Seqiences
Video #3: Generate Pose Asset with visemes (CC3)
Video #4: MetaHuman Setup (tutorial)
Video #5: Pose asset for MetaHuman from default visemes
New tutorial for MetaHuman.
New tutorial for CC3/CC4.
Feb. 24 update note: New language setup and packaging pipeline
Unlike text-to-lipsync solution, this is true voice-to-lipsync plugin. You don't need subtitles to get lips animatied, and resulted animation is much closer to speech then in case of subtitles-based solution.
Lip-sync can be generated in runtime, but not in real-time. I. e. it doesn't work with microphone or other streamed audio.
Fully supported languages: English, Chinese. Also supported: Russian, Italian, German, French, Spanish, Portuguese, Polish.
Whisper add-on (to use Whisper instead of Vosk): YnnkWhisperRecognizer
Included formats
- versions