What this actually is
The client publishes The Daily Sicha: a numbered English-language study piece released every day, roughly ten to eleven minutes when read aloud. Issue 040 opens on the laws distinguishing a shor tam from a shor muad in the Rambam; issue 042 turns on a verse from Tehillim. The prose is English. The vocabulary is not.
A daily publication cannot book a narrator daily. So the job was one voice, reproduced faithfully enough that a subscriber does not notice the handover, generating a fresh eleven-minute episode on demand.
Why this is harder than a voice clone
Three problems, and only one is the one people expect.
The lexicon fights the front end. Text is converted to phonemes by espeak-ng before the model sees it. espeak-ng’s English rules have no entry for Moshiach, teshuvah, farbrengen, ketores or Rambam, so it falls back to English letter-to-sound guesses. The corpus shows this directly: across the 119 training transcripts Mashiach appears 13 times, Torah 8, Rambam 5, mitzvah 3 — each rendered in an English phoneme inventory containing none of the sounds those words actually need. There is not one Hebrew character anywhere in the training data and not one guttural phoneme. The model is not learning Hebrew. It is learning to say English approximations of Hebrew consistently, in one voice, the way the human narrator does.
The corpus is a tenth of the specification. The intake spec below asked for 1.5–2 hours. What could be obtained was 12.4 minutes across 119 clips. That gap drives every later decision: full fine-tune rather than training from scratch, hard staging of the loss terms, and a pretrained checkpoint carrying most of the load.
The failure mode only appears over minutes. Style-vector wander, speaker identity drift, pacing collapsing toward monotone, the pitch contour resetting at each paragraph. None of it is visible in a ten-second sample, and an eleven-minute daily episode is nothing but the long tail of it.
Building the corpus
The intake spec
Corpus quality is decided before a single file is recorded, so Phase 2 of the project was a written specification issued to the speaker rather than a cleanup pass afterwards. Every line maps to a failure mode in the trainer:
| Requirement |
Why it is in the spec |
| 48 kHz audio, no additional noise |
Room tone and handling noise survive segmentation and get learned as part of the voice. The decoder cannot separate what it was never taught to treat as separate. |
At least 1.5–2 hours in a single .wav |
Below roughly this, style diffusion has too little variation to sample from and the voice flattens toward one delivery. |
| A matching transcript in text form |
Non-negotiable — the text side is what PL-BERT and the aligner consume. |
| 100–200 ms of silence at head and tail |
Gives segmentation a clean boundary to cut on, and stops the duration predictor learning that utterances begin abruptly. |
| Clear pronunciation, undisrupted across words, transitions, sentences and emphasis |
Disfluencies at a boundary produce alignments the model cannot explain, and the error propagates into the duration predictor. |
| Strictly one speaker |
The goal is one voice reproduced faithfully. A second voice anywhere contaminates the speaker identity. |
The spec was met on every axis except duration. Saying so plainly matters more than hitting the number, because it explains the training strategy that follows.
The SNR gate
Audio was not trusted because it sounded fine. Each candidate was converted to 16 kHz mono PCM with FFmpeg and scored with WADA-SNR — blind signal-to-noise estimation from the waveform amplitude distribution, no clean reference required. The notebook prints each file’s score beside an inline player, so a marginal score prompts a listen rather than settling the question.
This is the step most single-speaker corpora skip, and it is why 12.4 minutes was enough. A clean twelve minutes trains. A noisy hour teaches the decoder to reproduce the noise.
What came out
| Property |
Value |
| Clips |
119 — 108 train, 11 validation |
| Total duration |
12.4 min |
| Clip length |
median 5.4 s, range 1.6–14.0 s |
| Format |
24 kHz, mono, 16-bit PCM |
| Speakers |
1 |
| Transcript form |
espeak-ng IPA with stress marks |
| Minimum length |
50 frames; shorter clips dropped, never padded |
Transcripts are stored phonemised rather than as text — wav1-00099.wav|wˌʌts ðə bˈɪɡ dˈiːl hˈɪɹ ?|0. A separate pool of 141,433 phonemised LibriTTS train-clean-360 lines supplies out-of-distribution sentences for the adversarial stage, so the discriminator hears text far outside the twelve minutes of in-domain speech.
The stack, component by component
Every value below is read from the shipped configs, not from the paper.
| Component |
Configuration |
Why it is set that way |
| Mel front end |
24 kHz, n_fft 2048, win 1200, hop 300, 80 mels |
Inherited from the LibriTTS checkpoint; changing any of it invalidates the pretrained decoder |
| PL-BERT |
12 layers, 768 hidden, 12 heads, 178-token phoneme vocab, 1M pretraining steps |
Phoneme-level rather than word-level — the only reason an out-of-vocabulary loanword is representable at all |
| Style diffusion |
3 transformer layers, 8 heads, 64 head-features, mask prob 0.1, sigma_data 0.2 |
Samples the 128-d style vector instead of fixing it, which is what stops delivery flattening |
| Duration predictor |
max_dur 50, 3 layers, hidden 512 |
Rhythm and pause length; the term most sensitive to bad segmentation |
| F0 / energy |
JDC extractor, lambda_F0 1.0 |
Intonation contour. Drop this term and the voice goes monotone within a few epochs |
| Aligner |
ASR model, 80-mel in, 256 hidden, 178 tokens |
Supervises text–audio alignment; enters at epoch 10 |
| Decoder |
HiFi-GAN, upsample 10·5·3·2, 512 initial channels |
The upsample product is 300 — exactly the mel hop, so the decoder reconstructs precisely the frame rate it was handed |
| SLM discriminator |
WavLM base+, 13 layers, 16 kHz |
Adversarial naturalness signal, resampled down from 24 kHz |
| Optimiser |
lr 1e-4, bert_lr 1e-5, ft_lr 1e-4, weight decay 1e-4 |
PL-BERT is fine-tuned an order of magnitude slower than the acoustic model, deliberately |
Staging the fine-tune
The loss is a weighted sum, and the weights are not uniform: mel at 5.0, cross-entropy at 20.0, everything else at 1.0. The CE weight is what keeps the duration predictor honest on a corpus this small.
More important than the weights is the schedule. Three terms switch on at fixed epochs:
- Epoch 10 — monotonic alignment. Nothing downstream means anything until text and audio are aligned.
- Epoch 20 — style diffusion. Only once alignment is stable is there a meaningful style to sample.
- Epoch 30 — joint objective and SLM adversarial. The stage that buys naturalness, and the one that destroys an unconverged model if it arrives early.
The adversarial stage is itself throttled: 2% of batches, every 10 iterations, only on sequences between 400 and 500 frames, at scale 0.01 with a divergence threshold of 5. Those are guard rails, not tuning knobs — they exist so a discriminator that starts winning cannot drag the generator down with it.
The staging is visible in the training screenshot: Sty Loss: 0.00000, Diff Loss: 0.00000, SLoss: 0.00000 at epoch 1 is not a bug, it is the schedule.
What the run actually did
Two runs, both on 10 January 2025, both logged in full.
|
First stage |
Second stage |
| Epochs |
200 |
100 |
| Steps per epoch |
36 (batch 3 × 108 clips) |
36 |
| Validation loss |
0.895 → 0.555 |
0.416 → 0.370 |
| Best validation |
0.542 at epoch 189 |
0.361 at epoch 68 |
| Active terms |
mel only |
all, per the schedule |
The second stage ran 6 hours 9 minutes wall clock, 03:27 to 09:36. Duration loss settled at 0.87–0.88 and F0 loss at 1.97–2.05, both flat for the final thirty epochs — the signal that more training was buying nothing.
Judging it
The loss stops being informative early. By epoch 40 the validation curve is flat to within noise while the audio is still audibly improving in ways the mel term does not price: consonant crispness, the pause before a clause, whether a loanword lands the same way twice. So checkpoints were selected by listening, on held-out sentences, A/B against the previous checkpoint from a notebook harness. TensorBoard’s 118 MB event file is almost entirely logged audio previews, for exactly that reason.
Where a number is wanted, the standard no-reference tools apply — UTMOS for predicted MOS without assembling a listening panel, NISQA for per-dimension quality, whose discontinuity axis is the one that catches long-form seam artefacts.
One honest finding. Checkpoints were written every 50 epochs, producing epoch_2nd_00049.pth and epoch_2nd_00099.pth. The best validation loss of the whole run — 0.361 — landed at epoch 68 and was never saved. The shipped checkpoint scores 0.370. The gap is inside listening noise and did not change the delivered result, but the save interval should have been tighter, and on a rerun it would be.
The acceptance test was long-form. Short samples flatter a TTS model. Every failure mode that matters here — style vector wandering, speaker identity shifting, pacing collapsing into monotone, pitch contour resetting each paragraph — needs minutes to surface. So the test was to generate a full episode in one pass and listen to all of it: eight minutes forty-one seconds, continuous, from a single run. Not a montage of clips concatenated afterwards, which is what a demo reel usually is and which proves nothing about stability.
Code-switching
Success here is not whether Moshiach is pronounced correctly in a linguistic sense — espeak-ng’s English G2P cannot produce the right sounds, and no amount of fine-tuning repairs the front end. It is whether the word is pronounced the same way every time, at the prosodic weight the narrator gives it, without the English sentence around it losing its rhythm. That is a consistency property of the style vector and the duration predictor, and it is what the 119 clips were selected to teach.
Where it was going
The pretrained checkpoint is 771 MB; the fine-tuned one is 2.25 GB because it carries optimiser state, so stripping to weights-only is the first step toward serving. After that: a pronunciation lexicon patching the recurring transliterations ahead of espeak-ng, which is a far cheaper fix than more audio — and a rerun with a tighter checkpoint interval and the corpus at the duration the spec originally asked for.