ML

Flow Models for Audio & Speech Synthesis

Applying normalizing flows and flow matching to audio generation: WaveGlow, Glow-TTS, VoiceFlow, Matcha-TTS, and the advantages of flows for real-time speech synthesis with exact likelihood.

Advanced

Prerequisites

Table of Contents

  1. Learning Objectives
  2. Notation
  3. Core Intuition
  4. WaveGlow: Flow-Based Waveform Generation
  5. Glow-TTS: Flow for Alignment
  6. Flow Matching for Speech (Matcha-TTS)
  7. VoiceFlow & Voice Conversion
  8. Advantages for Real-Time Synthesis
  9. Common Pitfalls
  10. Summary
  11. Exercises

Learning Objectives

  1. Explain why flows are natural for audio (exact likelihood, fast parallel synthesis).
  2. Describe WaveGlow's squeeze + coupling architecture for waveforms.
  3. Explain how Glow-TTS uses flows for text-speech alignment.
  4. Describe flow matching for TTS (Matcha-TTS).
  5. Compare flow-based vs autoregressive vs diffusion-based TTS.

Notation

  • x∈RT\mathbf{x} \in \mathbb{R}^T — audio waveform (time samples)
  • mel∈RF×L\mathbf{mel} \in \mathbb{R}^{F \times L} — mel-spectrogram conditioning

Core Intuition

Audio synthesis has strict requirements: real-time speed, high fidelity, and controllability. Flows offer: (1) exact likelihood for stable training, (2) parallel synthesis (unlike autoregressive WaveNet), and (3) latent space manipulation for voice control. Modern flow matching TTS (Matcha-TTS) achieves near-autoregressive quality with 10x speed improvement.

Interactive: Normalizing Flow Transformation

z ~ N(0, I)

x = f(z)

Flow type:

Key idea: Normalizing flows transform a simple base distribution (Gaussian) into a complex target through invertible mappings. More layers = more expressive. The change-of-variables formula tracks the density through each layer.

WaveGlow: Flow-Based Waveform Generation

Architecture (Prenger et al., 2019): Coupling-based flow conditioned on mel-spectrogram:

xaudio=f−1(z;mel),z∼N(0,I).(1)\mathbf{x}_{\text{audio}} = f^{-1}(\mathbf{z}; \mathbf{mel}), \quad \mathbf{z} \sim \mathcal{N}(0, \mathbf{I}). \tag{1}

Key design:

  1. Squeeze: Group 8 consecutive audio samples into channels (reduce temporal, increase channel).
  2. Affine coupling: Conditioned on mel-spectrogram via WaveNet-like dilated convolutions.
  3. Invertible 1x1 conv: Mix channels between coupling layers.
  4. 12 coupling layers total.

Training: Maximum likelihood on paired (audio, mel) data.

Inference: Parallel synthesis — all time steps generated simultaneously. 500x faster than WaveNet.

Glow-TTS: Flow for Alignment

Key innovation: Use a flow to learn text-speech alignment without external aligner:

p(mel∣text)=pZ(f−1(mel))⋅∣det⁡Jf−1∣.(2)p(\mathbf{mel}|\text{text}) = p_Z(f^{-1}(\mathbf{mel})) \cdot |\det\mathbf{J}_{f^{-1}}|. \tag{2}

Alignment mechanism:

  1. Text encoder produces hidden states h1,…,hN\mathbf{h}_1, \ldots, \mathbf{h}_N.
  2. Monotonic alignment search (MAS) finds the optimal text-mel alignment.
  3. Flow maps each aligned segment from latent Gaussian to mel-spectrogram frames.

MAS (dynamic programming):

A∗=arg⁡max⁡A∑j=1Llog⁡N(f−1(melj);μA(j),σA(j)).(3)A^* = \arg\max_A \sum_{j=1}^L \log\mathcal{N}(f^{-1}(\text{mel}_j); \mu_{A(j)}, \sigma_{A(j)}). \tag{3}

Advantage: No need for pre-trained aligners (like Montreal Forced Aligner). Alignment learned jointly with synthesis.

Flow Matching for Speech (Matcha-TTS)

Matcha-TTS (Mehta et al., 2024): Apply optimal-transport flow matching to TTS:

L=Et[∥vθ(xt,t,text)−(mel−x0)∥2].(4)\mathcal{L} = \mathbb{E}_t \left[\|v_\theta(\mathbf{x}_t, t, \text{text}) - (\mathbf{mel} - \mathbf{x}_0)\|^2\right]. \tag{4}

Architecture:

  • U-Net with 1D convolutions for mel-spectrogram prediction.
  • Text conditioning via cross-attention.
  • OT-CFM path for straight trajectories.

Sampling: 2-10 Euler steps (vs 100+ for diffusion TTS).

Quality: Comparable to Grad-TTS (diffusion) with 10x fewer steps. MOS scores within 0.1 of ground truth at 4 steps.

VoiceFlow & Voice Conversion

Voice conversion: Transform one speaker's voice to another while preserving content.

Flow approach: Learn a flow in speaker-conditioned latent space:

  1. Encode source speech to content representation.
  2. Flow transforms content + target speaker embedding → target mel.
  3. Vocoder converts mel to waveform.

Advantage of flows: The latent space is well-structured (Gaussian), enabling interpolation between speakers, emotion control, and style transfer via latent manipulation.

Advantages for Real-Time Synthesis

  • Parallel generation: All mel frames generated simultaneously (unlike autoregressive).
  • Exact likelihood: Stable training, no mode collapse.
  • Few-step with flow matching: 2-10 NFE for high quality.
  • Latent control: Smooth latent space for interpolation and editing.
  • Deterministic: Same noise → same output (reproducible).

Common Pitfalls

Pitfall 1. Using too many flow steps in WaveGlow for real-time streaming. 12 coupling layers with WaveNet subnetworks are expensive; optimize network size for target latency.

Pitfall 2. Expecting flow-based vocoders to match GAN vocoders (HiFi-GAN) in raw quality. For waveform synthesis, adversarial training still produces sharper results.

Pitfall 3. Not conditioning the flow on sufficient speaker information. For multi-speaker TTS, speaker embeddings must be injected at multiple levels (not just globally).

Summary

  • WaveGlow: Coupling flows for parallel waveform synthesis; 500x faster than WaveNet.
  • Glow-TTS: Flow for joint alignment learning; no external aligner needed.
  • Matcha-TTS: OT flow matching for 2-10 step high-quality mel generation.
  • Voice conversion: Flow latent space enables speaker/style manipulation.
  • Flows excel at speech due to: parallelism, exact likelihood, few-step quality.

Exercises

Exercise 1. Compute the real-time factor (RTF) for WaveGlow generating 22kHz audio with 12 coupling layers on an A100 GPU.

Exercise 2. Derive the MAS algorithm (dynamic programming) for Glow-TTS alignment.

Exercise 3. Compare the number of model evaluations for Matcha-TTS (4 Euler steps) vs Grad-TTS (100 diffusion steps) for generating a 5-second utterance.

Exercise 4. Design a flow-based voice conversion system: specify the content encoder, speaker encoder, and flow architecture.

Exercise 5. Explain why flow-based TTS produces less "metallic" artifacts than GAN-based TTS (hint: likelihood training vs adversarial training dynamics).

Next

Rectified Flows & Reflow