Luna-TTS Family 🌙

Diffusion-Language-Model-Based Text-to-Speech with Fully Parallel and Streaming Generation

VUI Labs Research

Technical Report: arXiv:2608.11593

Abstract: Autoregressive codec language models decode speech left to right, which costs latency that grows with utterance length, error accumulation along the committed prefix, and a generation order the RVQ token grid does not have. The Luna-TTS Family is a set of diffusion-language-model-based TTS systems pretrained on 1 million hours of speech across Chinese, English, Japanese, and Korean, built by progressive adaptation of a pretrained autoregressive text LLM—causal to bidirectional to block-causal—over a shared 0.6B backbone. Luna-TTS generates the entire token grid in a fixed number of parallel refinement steps, with zero-shot voice cloning and speech editing arising natively as infilling. Luna-TTS Realtime is autoregressive over 1.28 s blocks while denoising each block in parallel, delivering audio incrementally once the first block is committed. An annealed fine-tuning stage adds emotion and non-verbal vocalization control, and a reinforcement-learning stage applies GRPO over the realized denoising trajectory. On Seed-TTS-Eval, Luna-TTS leads all four metrics among the systems we compare, at 0.6B parameters, while Luna-TTS Realtime returns its first audio in 41.6 ms (benchmarks below). All samples below are unedited model outputs at 24 kHz.

Contents

• Benchmark Results
• Zero-shot Voice Cloning
• Hard Text and In-the-wild Prompts
• Emotion Control
• Non-verbal Vocalizations
• Speech-rate Control
• Speech Editing
• Bonus
• Disclaimer

Benchmark Results

Zero-shot quality

On Seed-TTS-Eval — 2,020 Mandarin and 1,088 English utterances scored with the official toolkit — Luna-TTS is first in every column. Mandarin CER (0.73) leads MiniMax-Speech (0.83) and the same-protocol Qwen3-TTS baseline (0.98); English WER (1.49) leads Qwen-Audio-3.0-TTS (1.54). Speaker similarity is the highest in both languages. The streaming variant gives up roughly 0.3 CER/WER and 3 SIM points for incremental delivery, and still edges Seed-TTS and CosyVoice 3 on Mandarin CER — at 0.6B parameters, against systems two to three times its size.

^◇ from the system's own report  ·  ^† re-evaluated by us under the same protocol  ·  best per column in bold.

Inference performance

Luna-TTS returns the whole waveform at once, so its first audio is its full response; Luna-TTS Realtime commits one block at a time and plays before generation has finished. Running classifier-free guidance on a second GPU cuts full-response latency by about 45% and first audio from 59.6 to 41.6 ms. Luna-TTS at 16 steps is the throughput choice — a complete waveform in 216 ms; Realtime is the latency choice, first audio in 41.6 ms and a 10.6-second utterance finished in 254 ms.

BF16, batch 1, medians over 12 warmed runs on NVIDIA H20. First audio = engine invocation to the first decoded 0.36 s block.

Zero-shot Voice Cloning

Reference prompts are 3–10 s clips from speakers unseen during training. Outputs are generated by Luna-TTS.

Hard Text and In-the-wild Prompts

Two difficulties at once. The text is long, irregular, and stylistically diverse — tongue twisters, dense numerals, and expressive punctuation — where length heuristics and unconstrained stopping typically fail; the token-level duration predictor keeps the target length calibrated. The reference clips are in-the-wild recordings rather than studio takes, with uncontrolled channel, noise, and recording conditions carried over from the source corpus.

Emotion Control

Emotionally neutral prompts provide speaker identity. An utterance-level emotion tag and semantically aligned text specify the target emotion. Prompt and target content are different.

Non-verbal Vocalizations

Inline tags realize non-verbal vocalizations at the tagged position, alongside the surrounding speech rather than spliced into it.

Speech-rate Control

Fully parallel generation fixes the frame count up front, so rescaling the predicted duration gives continuous rate control — same voice, no retraining.

Speech Editing

Any-order infilling regenerates the highlighted span under the edited text, coherent with both past and future audio — no bespoke editing model.

Bonus

Reference audio here was collected from publicly available material online, and the lines are written for this page — nobody in the table ever said them. The first four rows take their reference audio and their text from the Seed-TTS demo page.

Disclaimer

The content provided above is for academic purposes only and is intended to demonstrate technical capabilities. Some examples are sourced from the internet. If any content infringes on your rights, please contact us to request its removal.