Skip to content

ADR-0013: Dictation through a Whisper program

  • Status: accepted
  • Date: 2026-09-25
  • Status update (Saturday, September 26, 2026): Implemented for audio files. Microphone dictation is still not available: cpal is only in the build as a dependency of rodio, and nothing records with it. As built: openai-whisper needs ffmpeg on PATH to decode audio; for whisper.cpp, a file is converted with ffmpeg unless its name ends in .wav; a program named main counts as whisper.cpp; and the spoken commands also accept "next paragraph", "next line", "newline", "fullstop", "semi colon", "begin quote", "end quote", "unquote", "left paren", and "right paren".
  • Status update (Saturday, September 26, 2026): The plan is to move Whisper into the process with candle, keeping this subprocess design as the fallback, and to measure its latency on the CPU.
  • Status update (Saturday, September 26, 2026): Whisper now runs in-process on RTen instead of candle; see ADR-0023. This subprocess design is the fallback. Microphone capture is rodio's recording feature (MicCapture), resampled with rubato; tw dictate without --file records until Enter when the in-process model is installed.

Context

star offered voice input two ways: "Transcribe Audio" opened a Whisper transcript of an audio file as a document, and "Dictate Note" recorded from the microphone until the student pressed Enter (Escape cancelled) and attached the transcript as a note at the reading position (star/transcribe.py, star/gui/mixin_transcription.py, star/tui/mixin_transcription.py). It paused the reading voice before opening the microphone, so Whisper would not transcribe star's own speech. It ran Whisper in-process through Python (openai-whisper or faster-whisper), recorded with sounddevice and numpy, and offered the model sizes of WHISPER_MODELS (tiny, base, small, medium, large-v3, large-v3-turbo; default base). It had no spoken commands; whisper_chunk_seconds was a dead setting.

textweaver is a Rust program. Linking a speech recognizer is heavy (whisper.cpp through bindings pulls a C++ build and, for speed, GPU back ends), and the machines students use differ: some have a Python Whisper already (this development machine has openai-whisper 20250625 in its Python 3.13 user site, with the tiny and base models cached), others can install a single whisper.cpp binary.

Decision

  • A Whisper program, run as a subprocess. textweaver-dictation runs one of three command-line programs, detected in this order: whisper.cpp's whisper-cli (older builds whisper-cpp), whisper-ctranslate2 (faster-whisper's command line), and whisper (openai-whisper's). TEXTWEAVER_WHISPER names a program explicitly (with TEXTWEAVER_WHISPER_ENGINE when its file name does not tell the engine). Nothing is installed or downloaded by textweaver.
  • Output. Each program prints one line per finished segment ([00:01.000 --> 00:03.500] text), which becomes a Partial event as it arrives, and writes a JSON transcript (whisper.cpp's transcription list, or openai/faster's segments list), which becomes the Final event; if no JSON appears, the printed segments are the transcript. Python programs run with PYTHONIOENCODING=utf-8. On Windows the process gets CREATE_NO_WINDOW, so no console flashes (a problem star documented).
  • Models. The model sizes are star's WHISPER_MODELS, default base. The Python programs fetch and cache their own models. whisper.cpp needs ggml-<model>.bin, looked up in TEXTWEAVER_WHISPER_MODELS, the app's data folder whisper/, and models/ beside the program or one level up; a missing model fails before recording starts, with the file name and the folders searched, so a student never talks into a session that cannot finish.
  • Audio. whisper.cpp reads 16 kHz WAV; other formats are converted with ffmpeg when it is on PATH (otherwise a clear failure). The Python programs decode any format themselves. Recorded audio is written as a 16-bit mono WAV in a per-session temporary folder, which is removed when the session ends. Audio never leaves the computer.
  • The trait. Dictation has start(input), stop(), cancel(), a non-blocking poll(), and state(), the shape of the speech backends (ADR-0003). Input is a file or an AudioCapture (start, stop returning PCM, cancel, elapsed). A session reports Recording, Transcribing, Partial per segment, then exactly one of Final, Failed (a message written for the student), or Cancelled. stop ends a recording and starts transcription (star's Enter); cancel discards a recording or kills the Whisper process (star's Escape) and reports Cancelled at once; a generation counter drops anything the old worker still sends. Each event has an announcement ("Recording", "Transcribing, this may take a while", "Transcribed 12 words", "Dictation produced no text", "Dictation cancelled").
  • Microphone. Capture needs an audio input library; cpal is not in the workspace, so this wave ships the AudioCapture trait and a BufferCapture, and tw dictate without --file explains that microphone dictation is not available yet. Adding cpal to the workspace is requested; a CpalCapture (16 kHz mono i16, resampled if the device cannot) then plugs in without other changes. The app must pause the reading voice before starting a capture, as star did.
  • Spoken commands are a pure transform on the finished text, apply_spoken_commands: "new line", "new paragraph", "period" / "full stop", "comma", "question mark", "exclamation mark/point", "colon", "semicolon", "ellipsis", "open/close quote", "open/close paren(thesis)", "hyphen", "dash"; "literal" before a command word types the word. Whisper punctuates by itself, so the transform drops the punctuation Whisper attached to a command word and replaces punctuation Whisper put just before one ("world, period." gives "world."), and capitalizes after a sentence end or line break. It is applied to finished text, not partial segments, and is optional (tw dictate --commands), because a command word can also be meant literally ("the period of history").
  • Timestamps. Transcript::text_with_timestamps writes star's transcribe_timestamps format: [mm:ss] text per segment, [hh:mm:ss] from an hour on.

Consequences

  • No Python and no native recognizer build are needed to build textweaver; dictation works wherever one of the three programs is installed, and whisper.cpp alone (one binary plus one model file) is the lightest route for students.
  • Start-up cost is the program's: on this machine openai-whisper with the tiny model took 8.3 s for 5 s of synthesized speech with a warm cache, and 127 s on the first cold run (Python and PyTorch loading). whisper.cpp should start in well under a second; that is to be measured when it is installed. Partial events arrive per segment, not per word, so live feedback during a long file is coarse.
  • Tests use a fake Whisper program (tw-fake-whisper) that speaks each engine's command line; a real-program test is ignored unless TEXTWEAVER_WHISPER_REAL=1 (it passed here against openai-whisper tiny).
  • True streaming dictation (words appearing while the student speaks) would need a streaming recognizer or chunked re-transcription; it is left for a later wave.

See also