Talking video quality: why lip sync is hard and how to judge it

People are experts at faces long before they are experts at anything else. We read lips without knowing it, and we notice within a fraction of a second when mouth and sound do not agree. That is why talking video is one of the hardest problems in AI video and why small technical choices make a large difference. This article explains what lip sync has to get right and how the talking video in AI Influencer Studio approaches it.

Sounds and mouth shapes are not one to one

Speech is made of sounds (phonemes); the visible mouth positions are called visemes. Many sounds share one viseme: p, b and m all close the lips. Other differences are invisible: k and g happen at the back of the mouth. A lip sync system has to produce the right sequence of mouth shapes for the sound, not a shape per letter.

Real mouths also anticipate. The lips round for the "oo" in "food" before the f is finished. This overlap, called coarticulation, is why speech looks fluid, and why a mouth that snaps from shape to shape looks robotic even when every individual shape is correct.

Timing tolerance is small

Viewers tolerate sound that lags the picture slightly better than sound that leads it, but the window is narrow, roughly a few video frames. Errors that stay inside that window go unnoticed; errors that build up during a long sentence become obvious. This is one reason long speech is split into shorter clips rather than generated in one piece.

The face has to be visible

Lip sync needs a mouth to work with. A small face in a wide shot, a strong profile, a hand or a cup in front of the mouth, or fast head movement all make good results less likely. Scenes made for talking, with the face large and turned towards the camera, consistently look better. When no face is found at all, falling back to a voice-over is better than forcing it, and the lip sync cost should be returned.

Language matters

Languages differ in speaking rate, rhythm and typical mouth shapes. A sentence of the same length takes different time in Turkish, German or Italian, which is why the length of a talking clip is calculated from the text and the language rather than a fixed rate. Mixing languages in one line makes both pronunciation and mouth movement less reliable.

Two approaches: audio first or video first

Audio first: generate the voice from text, then animate a face to match the audio. The voice is exactly right and the text is exactly what was written. The weakness is the face: animating an existing portrait to a fixed audio track often moves the mouth well but the rest of the face, head and body less naturally.

Video first: generate the character speaking, with speech, in one pass. Mouth, face, breathing and gestures belong together, so it looks more natural. The weaknesses are control: the voice is not your character's voice, and the words may differ slightly from your text.

The high-quality talking video combines the strengths. The clip is generated video first, so movement is natural. The speech is then converted to the character's saved voice while keeping its timing, so the mouth still matches. Finally the spoken words are transcribed and compared with the script; a clip that does not say the text closely enough is made again, and if that fails, the video is finished audio first as a backup, with the price difference refunded. The talking video guide covers the user-facing side.

How to judge a talking video

• Closures: do the lips close fully on p, b and m?
• Timing: watch the first and last words of each sentence, where drift shows first.
• Whole face: do the eyes, cheeks and head move with the speech, or only the mouth?
• Joins: in longer videos, are clip transitions smooth in picture and sound?
• Accuracy: does the video say exactly your text, with names pronounced correctly?
• Voice: is it the same voice from the first clip to the last?

What still limits quality

Very long single sentences, fast speech, singing, strong emotional shouting and several people speaking on camera at once remain difficult. Square videos and longer lines in scenes use standard lip sync rather than the high-quality method. And every check that improves quality also adds time: a talking clip takes about two minutes to make.

Frequently asked

Why not lip-sync any existing video? Lip sync on an existing clip only changes the mouth. When the rest of the face was not moving with speech, the result looks dubbed. Generating the clip as speech gives a more natural face.

Why are talking clips at most eight seconds? Shorter clips keep timing and identity tighter. Longer text is split at sentence boundaries and joined with short, soft transitions.

Does the character's own voice survive the process? Yes. The voice saved with the character, from the voice library or your own cloned voice, is the voice you hear in every clip.