Subtitles in time with the voice

Warning

Rod and Reef hasn’t been released yet, and videos or GIFs of this system can’t be shown until it launches.

Everything spoken in Rod and Reef is subtitled. In the protagonist’s lines, each word appears exactly when it’s said, with the pauses where the actor paused.

How they were made

Syncing every word of hundreds of lines by hand wasn’t an option, so I wrote a program that does it on its own:

• It listens to each clip with OpenAI Whisper, a speech recognition model, and notes when each word starts.
• It figures out which script line each recording is, even when the actor improvised or swapped words.
• It writes the subtitles ready for the game, using what was actually said rather than what the script said.
• It uploads the audio to Roblox and connects each subtitle to its clip.

If I change how the timing is calculated, I regenerate everything in seconds without listening to the audio again.

In the game

• Text is typed by following a clock, so it doesn’t run ahead or fall behind even if the game hitches for a moment.
• Several lines can be on screen at once without covering each other.
• In cutscenes, the subtitle follows the audio’s position every frame, so it’s always in sync.
• You can turn subtitles off in the settings without turning off the voices.

The details, including the timing formulas, are in the portfolio version.