Subtitles that wait for the voice.

Warning

Rod and Reef hasn’t been released yet, and videos or GIFs of this system can’t be shown until it launches.

Abstract. Rod and Reef has recorded voice in three formats: 74 short lines from the protagonist (the ambient barks), long cutscene dialogues and the monologues of a secret level, the Dream Maze. All of it is subtitled, and in the barks each word appears when the actor says it. Syncing the 888 words of the barks by hand, plus everything else, wasn’t viable, and the actor improvised, so the script couldn’t be the source of the text either. This article describes the pipeline I built to generate them. One phase transcribes every clip with faster-whisper large-v3 on the GPU and keeps the timing of every word. Another works out which script line each take is with fuzzy matching restricted by category. A third writes one Luau module per clip with each word’s duration and pause, anchored to the moment it’s spoken. The last uploads the audio to Roblox. In game, the player reveals words by following a clock rather than chained waits, so it doesn’t accumulate drift.

1. The problem

Barks are one-to-seven-second lines the protagonist says while you play (they are described in another article in this portfolio). A subtitle that appears all at once is read before the actor finishes the first word, and one typed at a constant speed runs ahead of or behind the take depending on its rhythm. What I wanted was for each word to start appearing exactly when it’s said, with the actor’s pauses where the actor paused.

That takes a start time per word. The barks alone are 888 words, and the cutscenes and the Dream Maze add many more. On top of that, the script wasn’t a reliable source: the actor improvised, swapped words, stretched some lines out enormously, and the document wasn’t even consistent with itself. The first three sections numbered their lines locally (1 to 9, 1 to 8 and 1 to 9), and from the fourth section on the numbering was global, 33 to 100, with a dead gap between 27 and 32. So the script could only be used to identify which line each clip was, and the subtitle text had to come from what was actually said.

2. The shape of the pipeline

The pipeline has five phases. Each writes its results to an output folder and can be re-run on its own:

Parse the script into a bank of lines per category.
Transcribe every clip with per-word timestamps.
Match each transcription to a line in the script.
Generate one Luau module per bark.
Upload the clips to Roblox and regenerate the modules with their asset ids.

Only phase 1 uses the GPU, and it’s the expensive one. Its raw output, a log with every word, its start, its end and its probability, is kept in full and never deleted. Changing the matching threshold or the timing formula doesn’t require transcribing again, because phases 2 and 3 run on top of that log. Transcription is incremental by default unless asked otherwise, and when a take has a re-recording (3 (v2).wav) the newest version wins.

3. Transcription

Transcription uses faster-whisper, a reimplementation of Whisper [1] on top of CTranslate2, with the large-v3 model on CUDA at 16-bit precision, a beam width of 5 and word timestamps. Two parameters differ from the defaults, each for a specific reason.

Conditioning on previous text is off. Every bark is a standalone line with no context, and carrying over the previous clip’s text is the classic recipe for Whisper hallucinating repetitions on short clips. Voice activity detection (VAD) is off too, because these are studio takes that are already trimmed, and VAD eats short interjections, like a “YEAHHH!!” on the boat, as if they were noise.

On Windows there was a problem that looked like hardware and wasn’t. CTranslate2 doesn’t ship the CUDA libraries inside its package: they come from the nvidia-*-cu12 packages, which put them in site-packages, and Windows doesn’t look for DLLs there. The first call failed with “cublas64_12.dll is not found”, which reads like a broken GPU. A small module adds those folders to the DLL search path before faster-whisper is imported, and with that it runs on an RTX 5060 Ti.

4. Which line is this clip

4.1 Search space

The clips were organized in folders by category (idle, walking, boat, fishing, jokes, reactions per NPC, special moments and the late game), and each folder maps to the matching section of the script. There are 14 of these scopes. Restricting the search to the lines in a scope, a handful instead of the document’s 94, is what makes fuzzy matching viable: with so few candidates, an improvisation still looks more like its own line than like any other.

4.2 One score for two kinds of improvisation

The actor improvised in two ways, and each one breaks a different score. Sometimes he reordered or swapped individual words. Other times he ran very long on a short line: “Let’s see…” in the script became seven seconds about cyclopes, sirens and an angry sea god.

Both scores come from RapidFuzz and are based on the edit distance with insertions and deletions only (Indel), a variant of Levenshtein’s [2]. For two strings $a$ and $b$ , the base similarity is

ratio (a, b) = 100 (1 - \frac{d_{Indel} (a, b)}{| a | + | b |})

token_sort_ratio sorts the words of both strings before comparing them, so it absorbs reordering while still penalizing missing words, which is what separates two similar lines from the same section. partial_ratio compares the shorter string against the best window of the same length inside the longer one, so it recognizes a script line as a stretch inside a long improvisation, exactly where token_sort_ratio sinks because of the length difference. Neither covers the other, so the score is the maximum:

s (t, g) = \max (t o k e n_s o r t (t, g), partial (t, g))

over the transcription $t$ and the script line $g$ , both normalized: no stage directions in parentheses, no punctuation and all lowercase.

partial_ratio tends to inflate the scores of short lines. A margin rule contains that: an automatic assignment needs a score of at least 80 and a lead of at least 8 points over the runner-up. If the inflated score favours the winner, it also favours the rivals, the lead shrinks and the case goes to manual review.

4.3 One to one

Two clips can’t claim the same line. Within each scope, every (clip, line) pair is scored, the pairs are sorted from best to worst and walked in that order: the first pair with both sides free is matched. It’s a greedy matching. It isn’t always optimal, which would call for the Hungarian method [3], but with scores as far apart as these it makes no difference, and it has one useful property: the extra clip doesn’t steal another clip’s line. The boat folder had 10 clips for 9 lines, and the tenth simply fell to review.

An empty transcription is the signature of a line with no words. Whisper found no words because there were none, and that’s a much cleaner signal than duration. The boat’s humming is matched that way to the only non-verbal line in its scope, before fuzzy matching can drag it anywhere, and its module ships with no text: the clip plays in full and no subtitle is drawn.

5. Word timing

5.1 The model

A bark is a one-node session, the whole line, split into one-word sub-nodes. Each word $i$ has a duration $d_{i}$ , the time the typewriter takes to reveal it, and a delay $δ_{i}$ , the stillness before the next one. If $T_{i}$ is the moment word $i$ starts appearing,

T_{i + 1} = T_{i} + d_{i} + δ_{i}

Whisper gives, for each word, the moment it starts being said, $s_{i}$ , and the moment it ends, $e_{i}$ . The text of each sub-node is the transcribed word, never the script’s.

The duration is cosmetic and comes from the word’s length, not from the audio, so no word pops in at once or drags on:

{\hat{d}}_{i} = clamp (0.03 | w_{i} |, 0.08, 0.35)

with $| w_{i} |$ the number of characters.

5.2 The first formula and its drift

The first version used the raw gap between words, $δ_{i} = s_{i + 1} - e_{i}$ . It looks natural and doesn’t work. With that formula,

T_{i + 1} - T_{i} = {\hat{d}}_{i} + (s_{i + 1} - e_{i}) while s_{i + 1} - s_{i} = (e_{i} - s_{i}) + (s_{i + 1} - e_{i})

so every word adds an error of ${\hat{d}}_{i} - (e_{i} - s_{i})$ , the difference between typing speed and how long the word lasts when spoken. Whisper chains almost every word end-to-start, with gaps of exactly 0.00, and nothing absorbs that difference. Summed over a line, the text finished about 1.7 seconds before the actor did. Scaling the gaps doesn’t help either, because most of them are zero, zero scaled is still zero, and all the leftover time would pile up in the two or three real pauses in the line.

5.3 Anchoring on the start

The version that stayed measures start to start. With $g_{i} = s_{i + 1} - s_{i}$ ,

d_{i} = \max (0.05, \min ({\hat{d}}_{i}, g_{i})), δ_{i} = \max (0, g_{i} - d_{i}), δ_{n} = 0

When $g_{i} \geq 0.05$ , the sum $d_{i} + δ_{i}$ is exactly $g_{i}$ , so $T_{i + 1} - T_{i} = s_{i + 1} - s_{i}$ , and summing,

T_{i} - T_{1} = \sum_{j = 1}^{i - 1} (s_{j + 1} - s_{j}) = s_{i} - s_{1}

Every word starts appearing exactly when it’s said, measured from the first one. It’s still the real gap from the audio; the only thing that changed is which pair of marks it’s measured between.

The cap $d_{i} \leq g_{i}$ is what makes that hold. Without it, a word spoken faster than its length suggests would get ${\hat{d}}_{i} > g_{i}$ , the delay would go negative, be clamped to zero, and the rest of the line would run late by that difference, a debt that only grows. The 0.05 s floor exists so a word never appears in a single frame, and it’s the only place debt remains: if $g_{i} < 0.05$ , the line falls behind by $0.05 - g_{i}$ . The total drift is bounded by

ε \leq \sum_{i : g_{i} < 0.05} (0.05 - g_{i}) \leq 0.05 \cdot # {i : g_{i} < 0.05}

which for normal speech is zero or close to it.

This is what a generated module looks like:

local BarkComedy01: Subtitles.SubtitleSession = {
	[1] = {
		Audio = "rbxassetid://112127624070584",
		Nodes = {
			{ Text = "If", Duration = 0.08, Delay = 0.2 },
			{ Text = "this", Duration = 0.12, Delay = 0.04 },
			{ Text = "fish", Duration = 0.12, Delay = 0.12 },
			-- ...
			{ Text = "lot.", Duration = 0.12, Delay = 1.18 },
			{ Text = "Repeatedly.", Duration = 0.33, Delay = 0 },
		},
	},
}

5.4 Only reviewing what needs listening to

Alongside the modules comes a report of divergences between transcription and script, computed with difflib’s SequenceMatcher, which implements Ratcliff and Obershelp’s algorithm [4]. From the code’s point of view, an actor’s improvisation and a Whisper mistake look the same. What the report does is flag the spans where the lowest word probability drops below 0.6 as likely transcription errors, and that cuts the list of things worth listening to with headphones down to a few.

6. Uploading to Roblox

Phase 4 uploads the clips through the Open Cloud API, owned by the game’s group and not by a personal account. Before uploading anything it checks every file against Roblox’s limits (20 MB, 7 minutes, 48 kHz and one, two, three or six channels) by reading the audio with PyAV, which already came along as a dependency of faster-whisper. Roblox’s documentation contradicts itself about the monthly upload quota, so the script warns against the lower figure, the only one that can surprise you halfway through a batch. The API key is read from an environment variable and never printed or written to disk.

The upload log is saved after every file, so an interrupted run resumes without paying again for what already went up. And the asset id isn’t patched into the Luau module as text. It’s kept in the log, and phase 3 regenerates the module by reading it. The generator is a pure function of the transcription log, the assignments and the upload log, so running it twice gives exactly the same output, and the result doesn’t depend on any fragile text replacement.

7. The player

7.1 Writing the text once

The SubtitleController writes the full line into the label once, before showing anything, and the only thing it moves afterwards is MaxVisibleGraphemes. Growing the text word by word forces the engine to lay the label out again on every word, and with centered text the line shifts under itself the whole time it’s being written. Before starting, every clip in the session is preloaded, so playback never races a sound that’s still loading.

7.2 A clock, not waits

The first version waited with task.wait between graphemes. A wait only resumes on the engine’s next heartbeat, so asking for 0.029 s actually costs about 0.037 s, a tax of roughly 25% that every grapheme of every word paid and accumulated. That’s why lines finished visibly after the voice had stopped.

The current player computes in advance, for each word, the grapheme it ends on, $G_{i}$ , and the moment it must be complete:

E_{i} = \sum_{j \leq i} k d_{j} + \sum_{j < i} δ_{j}

where $k$ is a global typing-speed factor that lets the writing speed be tuned by eye without regenerating a single module. It only scales durations, never delays, because the delays are measured silences from the performance, and stretching them would pull the line out of sync with the voice. Every frame it reads the elapsed time $τ$ and puts the label wherever the plan says it should be by now:

visible (τ) = G_{i - 1} + ⌊ n_{i} \cdot \frac{τ - (E_{i} - k d_{i})}{k d_{i}} ⌋

for the current word $i$ , with $n_{i}$ its graphemes, including the space before it. Words whose $E_{i}$ has already passed are caught up in a single write. A sampled clock can’t drift: the error is one frame at most, whatever the frame rate does, and after a hitch the text catches up instead of falling permanently behind.

When the last word has been typed, the player waits out whatever is left of the clip instead of cutting it, because the words set the pace, not the clip, and the actor sometimes keeps talking a moment longer. Then it holds the finished line for 1.25 s and fades it out over 0.5 s, the text and its outline together. That tail is exported as a constant, because anything sequenced to start some time after a subtitle has to subtract it. Without that, a 2.5 s pause before the next voice in the Dream Maze ended up a good deal longer.

7.3 Several lines at once

Several sessions can play at the same time, and each one takes a row: row 0 is the label authored in Studio and the others are clones 34 pixels below it. Each session claims the lowest free row, so the stack has no gaps when a long session and a short one overlap. The player’s subtitle setting hides the line, not the performance: the session still takes its row, loads its audio and keeps its timing, only the canvas isn’t shown. If turning subtitles off cancelled the sessions, it would also silence the voice, which is a different setting’s job.

8. Three kinds of audio, three formats

Barks are the case above: one clip is one line and a one-node session, and there is a script to match against.

Cutscenes are 45 to 100 seconds of dialogue between the protagonist and an NPC, in one continuous file, with no written script, so the transcription is the source of truth and goes straight to generation. The format isn’t a session either. It’s a flat table of the form [second] = "line", where the second is the absolute start of the first word of each Whisper segment, measured from the start of the clip. The game’s cutscene player reads the sound’s current position every frame and shows the last line that’s due. The audio is the clock, so there is nothing to keep in sync, and a clip that arrives late or a client that stalls for a second corrects itself on the next frame. The pipeline generates the first version of these tables, and they are then reviewed by hand.

The Dream Maze sits in between. Each floor has an 18 to 45 second monologue, like a cutscene, but it’s played by the SubtitleController as a session, like a bark. That means converting the clip’s absolute times into the chain of relative durations and delays a session understands, since a session has no clock of its own. A session starts when it’s called, not at second zero of the audio, so the first node is a silent one, with no text, that lasts exactly the initial silence $s_{1}$ . Each node’s delay is the one that lets the next node start at its own second:

δ_{node} = s_{next} - s_{last} - d_{last}

A line on screen has at most 10 words, and the breaks are found recursively at the longest pause in the span, so a sentence is never cut in half. One detail cost a one-word line on the first floor. Inside a sentence spoken in one breath, Whisper chains the words with gaps of exactly zero, so every break point ties, and taking the first one cuts at the earliest possible position. Ties are broken by balance: among equal gaps, the one that leaves two halves closest in size wins, minimizing $| i - n / 2 |$ . Whisper segments less than 0.6 s apart are merged before splitting, because in these takes the segmenter’s false breaks measure 0.00 s and real pauses run past 1.5 s. And a couple of words Whisper made up are removed by their stem and start time, because text alone isn’t enough when the same word appears correctly elsewhere in the take.

9. Limitations

In the barks, the first word starts appearing when the audio starts, not at $s_{1}$ , so the text runs ahead of the voice by the take’s initial silence. With trimmed studio clips that gap is small, and the Dream Maze, where it does matter, handles it with the silent node. Greedy matching isn’t optimal in general, and it works because the scores of these takes are well separated. In the cutscenes, the quality of the line breaks depends entirely on where Whisper decides to cut, which sometimes splits a sentence in half or glues two turns of dialogue together, and that has to be caught by reading the generated module. And low-confidence words still need someone to listen to them.

10. What I took away

Keeping the raw output of the expensive step is what made it possible to iterate on everything else, because the timing formula changed several times and the GPU never had to be touched again. The other lesson was about drift. The two versions that failed (raw gaps and chained waits) had the same flaw: they added up small errors with nothing to correct them. The ones that work anchor every value to an absolute reference: the start of each word in the generator, the clock in the player and the sound’s position in the cutscenes.

References

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey and I. Sutskever. “Robust Speech Recognition via Large-Scale Weak Supervision”. Proceedings of the 40th International Conference on Machine Learning, 2023.
V. I. Levenshtein. “Binary Codes Capable of Correcting Deletions, Insertions, and Reversals”. Soviet Physics Doklady, 10(8), 1966.
H. W. Kuhn. “The Hungarian Method for the Assignment Problem”. Naval Research Logistics Quarterly, 2, 1955.
J. W. Ratcliff and D. E. Metzener. “Pattern Matching: The Gestalt Approach”. Dr. Dobb’s Journal, July 1988.