DeepGamingAI's video: Realistic Facial Animations Of 3D Avatars Driven By Audio And Gaze Game Futurology 34

This is episode of the video series "Game Futurology" covering the paper "Audio- and Gaze-driven Facial Animation of Codec Avatars" by Alexander Richard, Colin Lea, Shugao Ma, Juergen Gall, Fernando de la Torre and Yaser Sheikh. PDF: https://arxiv.org/pdf/2008.05023.pdf Authors' Project Page: https://research.fb.com/videos/audio-and-gaze-driven-facial-animation-of-codec-avatars/ Game Futurology: This is a video series consisting of short 2-3 minute overview of research papers in the field of AI and Game Development. This series aims to ponder over what the future games might look like based on the latest academic research going on in the field today. Subscribe for more weekly videos! Abstract: Codec Avatars are a recent class of learned, photorealistic face models that accurately represent the geometry and texture of a person in 3D (i.e., for virtual reality), and are almost indistinguishable from video. In this paper we describe the first approach to animate these parametric models in real-time which could be deployed on commodity virtual reality hardware using audio and/or eye tracking. Our goal is to display expressive conversations between individuals that exhibit important social signals such as laughter and excitement solely from latent cues in our lossy input signals. To this end we collected over 5 hours of high frame rate 3D face scans across three participants including traditional neutral speech as well as expressive and conversational speech. We investigate a multimodal fusion approach that dynamically identifies which sensor encoding should animate which parts of the face at any time. See the supplemental video which demonstrates our ability to generate full face motion far beyond the typically neutral lip articulations seen in competing work: this https URL Music Credits: https://www.fesliyanstudios.com/ ---------------------------------------------------------------- • YouTube - https://www.youtube.com/c/DeepGamingA... • Twitter - https://twitter.com/deepgamingai • Medium - https://medium.com/@chintan.t93 • GitHub - https://github.com/ChintanTrivedi --------------------------------------------------------------------