Skip to content

ADR-0011: Audio export

  • Status: accepted
  • Date: 2026-09-25
  • Status update (Saturday, September 26, 2026): Implemented. DECtalk also gives exact word timings (ADR-0021). tw export-audio now reads the voice, rate, pitch, volume, preferred engine, table and footnote modes, and the [export] settings from settings.toml, and takes --home. As built: without --backend, the [speech] backend setting is used first when that engine can write files, then the best one that can; the result sentence names the engine and adds "Subtitles in FILE." when subtitles are written; and the per-sentence pieces go in the system's temporary folder.
  • Status update (Tuesday, September 29, 2026): Formats and tags without ffmpeg. FLAC is written in process with flacenc 0.5.1 (pure Rust, Apache-2.0; the export crate's default flac feature, and without it FLAC goes through ffmpeg). The joined WAV is encoded from a private folder beside the output, read in blocks; 8, 16, 24 and 32-bit PCM and 32 and 64-bit float are accepted, 32-bit PCM keeping its top 24 bits and float becoming 16-bit PCM. flacenc writes no tags, so the export adds a Vorbis comment block (metadata type 4) itself: TITLE, ALBUM, ARTIST, GENRE=Audiobook, and a CHAPTERnnn (start as HH:MM:SS.mmm) and CHAPTERnnnNAME pair per chapter. WAV files get the same title and chapters as an ID3v2.3 tag in an ID3 chunk, with one CHAP frame per chapter and one top-level ordered CTOC frame (the id3 crate, MIT; the default id3 feature). Both are read back in tests: the FLAC with the pure-Rust claxon decoder (a dev-dependency), which also shows the encoding lossless, and the WAV with id3. MP3 in process waits for a decision on LAME's LGPL license, so MP3 and M4B stay on ffmpeg as above. Also: the avspeech backend reports word times in exported files, from the synthesizer's word markers on macOS 14 and later, and a SAPI voice the engine host cannot apply now fails the export instead of leaving the host's previous voice to speak.
  • Status update (Tuesday, September 29, 2026, later): MP3 in process. With LGPL approved for it, MP3 is encoded with LAME 3.100 through mp3lame-encoder 0.2.5 and mp3lame-sys 0.1.11 (the export crate's default mp3 feature; without it MP3 goes through ffmpeg as before). We chose these crates because they are maintained, LAME is the reference MP3 encoder with the best quality at low bit rates, and mp3lame-sys builds LAME from its bundled, unchanged source with no system package: the cc crate with MSVC on Windows, LAME's own configure and make on Linux and macOS (present on the CI runners and in the container image). The pure-Rust shine-rs was passed over: fixed bit rate only, and lower quality. LAME's source is LGPL-2.0-or-later; both crates declare LGPL-3.0, which deny.toml and about.toml now allow. LAME is linked statically, so THIRD-PARTY-NOTICES.md states how to relink with a modified LAME (rebuild from source with a patched mp3lame-sys) and carries LAME's license text (third_party/lame/COPYING, also packaged under licenses/lame/). Encoding for speech: VBR (-V 5), mono for mono engines, the engine's sample rate, and a LAME (Xing) tag so decoders drop the encoder's delay and padding and report the exact length; the title, author and chapters follow as ID3v2.3 (CHAP and CTOC), as for WAV. Tests decode the MP3 back with the pure-Rust symphonia (MPL-2.0, tests only): within one MPEG frame of the timeline's length, chapters read back. The reader's Export audio flow offers MP3 without ffmpeg; M4B keeps ffmpeg, since no GPL-compatible AAC encoder runs in process.
  • Status update (Wednesday, September 30, 2026): Ogg Opus in process. .opus is encoded with libopus 1.6.1, the reference encoder, through opusic-c 1.6.1 and opusic-sys 0.7.5 (BSD-3-Clause, like libopus), built from the source bundled in opusic-sys with cmake and linked statically, with libopus choosing SIMD at run time; the export crate's default opus feature (without it, Opus goes through ffmpeg's libopus with the same settings). The owner approved libopus and cmake after the pure-Rust opus-rs 0.1.34 was measured and set aside: against libopus 1.5.2 as the decoder, its 24 kHz output was broken in every mode, it ignored the bit rate at 16 kHz (72 kbit/s for 24) and overshot it at 48 kHz in VoIP mode, and it lost a pure tone's level at 48 kHz in Audio mode (upstream issues 37 and 38; issue 30 is a crash on CPUs with AVX but no FMA). Encoding: the joined WAV is mixed to one channel and resampled to 48 kHz with rubato, then encoded in 20 ms packets in the Audio application at 32 kbit/s variable bit rate, complexity 10. The Audio application was chosen by measurement: on a 15-second recorded voice at 48 kHz, libopus gave 12.5 dB of waveform signal-to-noise at 32 kbit/s in Audio mode and 7.4 dB in VoIP mode, at the same size. The Ogg pages are written by the export crate (CRC, lacing, a page about every second): OpusHead with the encoder's look-ahead as pre-skip (312 samples) and the engine's rate as the input rate, OpusTags with the same Vorbis comments as FLAC (the shared vorbis module), and the last page's granule position trimming the padding, so a decoder returns exactly the engine's length at 48 kHz. Tests decode the file back with libopus: exactly the timeline's length, the chapters, and a resampled tone whose start, pitch, and level survive. The reader's Export audio flow offers Opus after MP3. Building now needs cmake for the default features; the workspace dependency on the export crate has default features off, and the reader (textweaver-app and textweaver-xilem) and tw each have an opus feature, on by default, to build without it.

Context

star exported audio three ways (the star parity reference, kept outside the repository, Part 2 ยง5.D; star/tts/audio.py, star/audiobook.py): the whole normalized text synthesized in one call to a WAV, then converted with ffmpeg (or pydub) to MP3, OGG, or MP4; an M4B audiobook synthesized chapter by chapter, joined by ffmpeg's concat demuxer with chapter metadata; and subtitles (star/tts/subtitles.py) built by spreading the whole file's duration over its whitespace tokens by length (len + 1), grouped into caption lines of at most 12 words and 90 characters, broken at sentences.

Three problems follow. Apportioning one duration over a whole document drifts: a long word, a pause, a number read as six words, or a heading announcement pushes every later cue away from its audio, and the error grows to minutes in a book. Engines that know exactly where each word sounds (espeak-ng's audio_position, ECI index marks, SAPI word-boundary offsets) had that knowledge thrown away. And export needed a backend-specific path per engine.

Decision

  • One pipeline over the backend trait. textweaver-export reads any document through any backend with Caps::SYNTH_TO_FILE. A new default trait method, SpeechBackend::synthesize_utterance(&Utterance, &Path) -> FileSynthesis, writes one utterance to a WAV file and returns word timings (WordTiming { byte_range, audio_ms }) when the engine knows them; its default calls synthesize_to_file and returns none. espeak-ng implements it from retrieval-mode word events; the recording test double writes deterministic silence (250 ms per word) with exact timings. ECI, SAPI, and DECtalk override it from their hosts' index marks and word-boundary events: the utterance is captured through the engine host, and each word's sample offset in that capture becomes its time in the file (textweaver_enginehost::word_timings), so the cue is exact on the file's own clock. SAPI voices without word timing (Code Factory's Eloquence) report none.
  • Utterance by utterance. The document is planned with text::narrate::plan exactly as for reading aloud (structure announcements included), each utterance is normalized with the same pipeline the speech service uses (skipping what an engine normalizes natively), synthesized to its own file in a private temporary folder, and appended to one output WAV. The running sample count gives each sentence its exact start and end, and each reported word its time, on the output's clock: the Timeline. An optional gap (default 0) can separate utterances. The WAV writer keeps the first piece's fmt chunk byte for byte and refuses a piece in another format; the RIFF sizes are patched at the end; 4 GB is the limit.
  • Subtitles from the timeline. star's cue rules are kept (12 words, 90 characters, breaks at sentences, len + 1 weighting, SRT and WebVTT formats, zero-length cues stretched to 50 ms, star's test vectors ported), but the weighting only spreads a sentence's own measured duration over its tokens, and tokens take engine word times when there are any. Captions show the document's text, not the normalized spoken text: word timings are mapped through each utterance's offset map, so "$5" read as "five dollars" is one word cue showing "$5", and spoken-only text ("heading level 1") has no caption. Word-level cues (star's subtitle_word_level) follow the engine's words when it reports them.
  • Chapters from structure. Every SectionBreak marker and every Heading up to a level (default 6, star's "every heading") starts a chapter at the time the reading first reaches it; a section that opens with a heading is one chapter titled by the heading; text before the first chapter is a chapter titled after the document (star's rule); chapters without audio of their own are dropped.
  • ffmpeg for MP3 and M4B, when present. ffmpeg is found through TEXTWEAVER_FFMPEG or PATH, never downloaded or bundled. The WAV and an ;FFMETADATA1 file (title, album, artist, genre=Audiobook, one [CHAPTER] per chapter in milliseconds) go to ffmpeg -map_metadata 1 -map_chapters 1: MP3 with LAME at VBR quality 2 (ID3 chapters included), M4B as 64 kbit/s AAC in MP4 with +faststart (star's settings). Without ffmpeg, MP3 and M4B fail with a sentence saying so before any synthesis starts; WAV always works. Intermediate files live in a temporary folder beside the output and are removed.
  • Command line. tw export-audio FILE --out out.wav|flac|mp3|m4b [--subtitles out.srt|vtt] [--word-level] [--backend --voice --rate --pitch] [--json] [--quiet]. Without --backend, the highest-priority available backend that can write files is chosen (engines that only play, such as Omnivox and speech-dispatcher, are passed over). Progress is reported in whole tens of percent, one short sentence each, so a screen reader is not flooded; the result is one sentence (length, sentences, chapters, voice). Cancellation between sentences is part of the library API for the GUI.

Consequences

  • Cue times no longer drift: every sentence cue starts and ends where its audio does, measured, and word cues are exact for engines that report word times.
  • Export costs one engine call per utterance. For engines with per-call start-up cost (host processes) this is slower than one large call, but it is what makes timing exact, bounds memory, and lets the user cancel.
  • Engines that cannot write files (Omnivox's write-only protocol, speech-dispatcher) cannot export; tw backends lists which can ("audio files").
  • Measured on 2026-09-25 on Windows with Microsoft David through SAPI: a two-heading sample exported to M4B, and ffprobe -show_chapters listed both chapters at the times of their headings.
  • Not done here: cover art (star read it from document metadata), OGG and MP4 targets, and video export with burned-in subtitles (star's video.subtitles = "burn", never implemented there either).

See also