AI Lip-Sync Workflow: Swap Voice and Match Mouths
Use an AI agent to swap the voiceover on an existing video and sync the mouths to it: drive dubbing and lip-sync in one natural-language chain for fast, scalable video localization.

Tool names in this guide are for reference only and may have changed or been renamed. Please use the latest tools in the tool marketplace when implementing.
Key takeaways
- Swap voice and match mouths in one chain — no frame-by-frame editing
- Drive dubbing and sync in natural language, localize many languages
- Verify naturalness before export to avoid the cheap "out-of-sync" look
Tools used
Why swapping the voice leaves the mouth out of sync
Teams doing multilingual or translated video all hit the same wall: the original footage is fine, you swap in a voiceover in another language for an overseas audience, and now the audio has changed but the mouth is still saying the original words. Viewers spot the mismatch instantly, and even great content starts to look cheap.
The traditional fixes are either hiring voice talent to time each line against the picture, or having an editor nudge the mouth frame by frame — slow, expensive, and hard to scale. The real difficulty isn't producing a new voiceover; it's getting that new voiceover to line up with the mouths already in the footage.
An AI agent connected to speech-synthesis and lip-sync tools folds this into one chain: you give it the original video and a new script, it generates the new voice, then syncs the mouths in the original clip to that voice, and outputs a localized video where audio and picture match.
The workflow in five steps
- Hand over the material: give the agent the original video and the new script.
- Generate the new voice: synthesize a voiceover from the script.
- Sync the mouths: align the mouths in the original video to the new voice.
- Verify naturalness: check that audio and picture line up and the mouths look natural.
- Ship the localized video: export, and batch the remaining languages when needed.
How to do each step
Step 1: Hand the agent the video and the new script
Prepare the original video file (or an accessible link) and the replacement script. Say what you want in natural language — which voiceover to swap in and what the target language is.
use ae to relocalize this video at example.com/clip with a new voiceover
Step 2: Generate the new voice
The agent calls a speech-synthesis tool to produce a voiceover from the script. For everyday text-to-speech use ae_f5_tts to turn the new script into a natural voice track.
use ae to generate the new voiceover from this script with ae_f5_tts
This step sets the timbre and pacing. Rewrite the script to the target language's spoken rhythm rather than translating word for word — it makes the later lip-sync fit better.
Step 3: Sync the mouths
With the new voice ready, use a lip-sync tool to align the mouths in the original clip to it, such as ae_latentsync. It anchors on the original footage and only rebuilds the mouth region tied to speech, keeping the rest of the frame as-is.
use ae to lip-sync the original video to the new voiceover with ae_latentsync
Step 4: Verify naturalness
Before exporting, do a pass: do audio and picture line up, do the mouths look natural at starts, stops and pauses, is there any obvious blur or jitter? This step is the line between "usable" and "looks like a native speaker." If it's off, go back to Step 2, adjust the script's pacing, and sync again.
Step 5: Ship and batch
Once one clip works end to end, apply the same flow to the other languages: same original video, different scripts — generate voice, lip-sync, verify, in turn. Compared with manual frame-by-frame work, this chain makes multilingual localization something you can scale.
Voice swap without lip-sync vs an AI lip-sync workflow:
| Swap voice only, no lip-sync | AI lip-sync workflow |
|---|---|
| New audio, old mouths — looks cheap | Mouths follow the new voice, audio and picture match |
| Lip-sync done frame by frame by an editor | One natural-language instruction drives the sync |
| Redo every language from scratch | Same chain batched across languages |
Tools by step:
| Step | Tool type | Tool (example) |
|---|---|---|
| Generate new voice | Text-to-speech / dubbing | ae_f5_tts |
| Sync the mouths | Lip-sync | ae_latentsync |
How to get started
- Sign up — new users get 1,000 free credits.
- Connect in-app from an agent like WorkBuddy or Trae by searching for "AgentEarth", or paste a one-line MCP config into Claude Code, Cursor or VSCode.
- Hand the agent the original video and the new script, and ask it to dub and lip-sync in natural language.
FAQ
Does it support multiple languages? Yes. Rewrite the script to the target language's spoken rhythm first, then generate the voice and lip-sync; the same chain handles one language after another.
Do the mouths look natural? The lip-sync tool anchors on the original footage and only rebuilds the speech-related mouth region. Verify naturalness at starts, stops and pauses before export, and if it's off, adjust the script's pacing and sync again.
Is there a length limit? Results depend on the length and pacing of the source video and voiceover, so run the flow on a short segment first to confirm naturalness, then extend to the full video.
