Overview

The constraint is the whole brief: the localized track has to keep the original timing, the original backing track, the original speaker's identity, and natural-sounding delivery — and it has to hold all four without a human ever nudging a single cut point by hand. Long-form video makes this unforgiving. A few hundred milliseconds of drift on one line is a minor, easily missed annoyance in a ten-second clip; the same drift compounded across a multi-minute clip is a track that's audibly, then visibly, off the picture. Nothing about a translation model, a voice-cloning model, or a text-to-speech model naturally respects the source's timestamps — each is trained to produce a natural-sounding utterance on its own terms, not one clipped to a slot someone else already defined. Everything below exists to impose that constraint on models that don't carry it by default.

Diarization and forced alignment

Before any translation or synthesis happens, the source audio is transcribed with WhisperX, which pairs a fast CTranslate2-backed Whisper model for the raw transcript with a forced-alignment pass that re-aligns each word against the audio using a phoneme-level acoustic model, rather than trusting Whisper's own segment boundaries. Whisper's native timestamps are a byproduct of autoregressive decoding and are notoriously loose — off by hundreds of milliseconds, sometimes more, especially around sentence boundaries. Forced alignment fixes that by scoring the audio directly against the known transcript text, phoneme by phoneme, and reporting the actual start and end of every word. Diarization runs alongside it, clustering speaker embeddings across the track so that every aligned segment also carries a speaker label — essential the moment more than one voice is on screen, since a segment with the wrong speaker attached gets the wrong voice cloned onto it several stages later.

The output of this stage isn't audio at all — it's a plain, inspectable list: speaker, start time, end time, and text, one row per phrase. Every later stage in the pipeline reads and writes against these timestamps, and nothing downstream invents its own; the timing skeleton is decided once, here, and never renegotiated.

Why not translate and synthesize without frame-level alignment

The naive version of this pipeline skips all of that: translate the transcript, synthesize the whole translation in one pass, and either stretch the single resulting clip to match the original's total runtime or just play it back sequentially, cut for cut. It fails almost immediately, because sentence-level duration doesn't translate consistently between languages. A phrase in the target language routinely runs shorter or longer than its source-language counterpart by a wide, unpredictable margin per line — driven by syntax and word choice, not anything the source clip's total runtime can tell you in advance. Concatenating segments of drifting length means the second line in a scene is already a beat behind, the fifth is noticeably behind, and by the second or third minute a viewer is hearing dialogue for a shot that has already ended. Anchoring only the start and the total length can make a short clip look fine by accident while every internal cut is still wrong — which is exactly why "frame-accurate" describes a target held continuously, at every segment boundary, not just at the two ends of the clip. Forced alignment matters here because it gives the pipeline a ground-truth target end time per segment to synthesize toward, rather than a running estimate it has to hope averages out.

Neural voice cloning

Once a segment's speaker, timing, and separately translated text are known, XTTS synthesizes that line in the target language in the source speaker's own voice rather than a single generic narrator voice. XTTS clones a voice from a short reference clip rather than requiring a trained per-speaker model, so the reference used for each segment is a clean excerpt of that same speaker pulled from elsewhere in the source track — diarization is what makes that possible, since it's the only reason the pipeline knows which earlier segment belongs to which of possibly several speakers on screen. Two people talking in the same scene come out of this stage as two distinct cloned voices, not one dub actor reading both parts.

Voice identity and timing are deliberately separate concerns at this stage. XTTS synthesizes each line at whatever duration is natural for that phrase in the target language — it has no concept of the slot it needs to fill, and giving it one would mean fighting the model for pacing at the cost of naturalness. The output is treated as provisional audio: correct in voice, correct in words, and not yet correct in length. Fitting it to the slot is a separate, later step, by design.

Source separation

Ahead of any synthesis, the original mix is run through Demucs, which splits it into an isolated vocal stem and an instrumental stem — music, ambience, sound effects, everything that isn't dialogue. This has to happen before, not after, the dub is composed, because the whole point is that the dub only ever replaces the vocal stem; the instrumental stem carries straight through to the final mix untouched, and the pipeline never resynthesizes or approximates the original score, room tone, or sound design. Rebuilding those from scratch, or trying to approximate them, would be both unnecessary work and a guaranteed quality loss compared to keeping the original bytes.

Source separation on a dense mix is genuinely hard, and Demucs isn't perfect at it — that shows up later, in the tradeoffs. But running it early, as an isolated, single-purpose step, means any imperfection in that step stays contained to the instrumental stem file it produces: inspectable and fixable on its own, rather than smeared across a monolithic re-render of the whole scene.

Time-stretching to fit the original slot

Forced alignment fixed where each segment belongs; voice cloning fixed what it should sound like. Neither one fixes how long the synthesized clip runs, and XTTS's natural output almost never matches the slot's width exactly. This stage closes that last gap: each synthesized segment is time-stretched — compressed or expanded — to exactly fill its aligned start-to-end window, without changing pitch, since a pitch-shifted clone stops sounding like the cloned speaker at all.

# synthesized segment ran 2.77s; its aligned slot is 2.35s wide
# stretch factor = synthesized_duration / target_duration = 1.18
ffmpeg -i segment_042.wav -filter:a "atempo=1.18" segment_042_fit.wav

Fitting one synthesized segment to its aligned slot

FFmpeg's atempo filter only accepts a bounded stretch ratio per pass, so a segment that needs a more extreme ratio gets there by chaining multiple atempo stages rather than one out-of-range call, and the filter itself uses a time-domain algorithm built to preserve pitch and timbre rather than naive resampling, which would otherwise shift pitch up or down with speed. The result is a clip of exactly the right duration for its slot, in the right voice, saying the right words — three properties that rarely arrive together on their own, without this step doing the reconciling.

End-to-end composition

Every earlier stage produces a plain, inspectable artifact rather than an opaque intermediate tensor: a list of aligned segments, a set of cloned-and-stretched voice clips, an isolated instrumental stem. Composition is where they come back together.

source.mp4
  -> demucs      vocal.wav, instrumental.wav
  -> whisperx    aligned segments (speaker, start, end, text)
  -> translate   target-language text per segment
  -> xtts        cloned speech per segment (provisional duration)
  -> stretch     speech fit to its aligned slot
  -> ffmpeg      stretched segments + instrumental.wav -> final mix -> mux against source video

The pipeline end to end

Each time-stretched segment is placed at the exact timestamp forced alignment assigned it, mixed against the untouched instrumental stem, and the result is muxed back against the original video stream — the picture itself is never touched. The framing that matters here is that the three main mechanisms correct three different, independent error modes rather than one mechanism trying to do everyone's job: alignment fixes where a line lands, time-stretching fixes how long it lasts once it's there, and source separation keeps everything that isn't the dubbed voice untouched and clean. None of the three papers over a mistake made by another; each is scoped narrowly enough to be individually correct, individually testable, and individually repairable if one segment comes out wrong, without re-running or re-trusting the other two. That's what lets a multi-minute clip hold its sync for its entire runtime instead of drifting the way naive concatenation does — nothing in the composition step is asked to compensate for drift, because nothing upstream introduces any.

Tradeoffs and limitations

• Time-stretching has audible limits. A translation that's unusually verbose relative to its source line still strains the slot, and past a certain ratio the pacing sounds unnatural well before any pitch artifact gives it away.
• Voice cloning quality is bounded by the reference clip. A noisy, heavily compressed, or overlapping source excerpt gives XTTS a lower-fidelity target, and the cloned voice drifts further from the original speaker than a clean reference would.
• Source separation isn't perfect. Demucs can leave audible bleed or artifacts in the instrumental stem on dense mixes — music under heavy dialogue is the hardest case — and whatever imperfection survives that step ships straight into the final mix.
• Diarization mistakes on overlapping or fast-alternating speakers propagate downstream. A misattributed segment gets the wrong voice cloned onto it, and no later stage is positioned to notice or correct that on its own.
• The pipeline optimizes timing, identity, and cleanliness, not translation quality itself. An awkward or imprecise translation is stretched onto schedule exactly as faithfully as a good one; nothing here judges whether the words are the right ones.

References and further reading

• WhisperX — timestamp-accurate speech transcription and alignment
• Coqui TTS — XTTS voice cloning
• Demucs — music source separation
• FFmpeg filters documentation
• Montreal Forced Aligner documentation