From Space to a spoken word

Step 1 of 10: Document

The document is canonical text in a rope, plus markers.

A rope is a tree of text pieces, so edits and lookups stay fast even in a very large book. Markers note what each range of the text is: a heading, a link, a list item, a table, code, or math. Every position is a character position, counted in Unicode scalar values. That keeps saved positions the same as star's. When you press Space, reading starts at the word under the cursor.

Decided in ADR-0002: Text model.

Step 2 of 10: Units

Words and sentences are found with the Unicode text segmentation rules, UAX #29.

An abbreviation list keeps "Dr." or "e.g." from ending a sentence. A few abbreviations, such as "a.m." and "etc.", end a sentence only when the next word starts with a capital letter. List items, table rows, and code lines are sentences of their own. The same units drive navigation, so "next sentence" and reading aloud always agree.

Step 3 of 10: Narration

The narration planner, narrate::plan, turns the text into sentence-sized utterances.

Continuous reading is planned in windows of about ten minutes of speech (32,768 characters). The next window is planned when this one ends, so a long book starts reading at once.

The planner adds spoken words for structure that has no text of its own. For example, it says "heading level 2" before a heading, and "list with 3 items" before a list. These are inserted spans: you hear them, but they have no place in the document.

Decided in ADR-0005: Narration and the OffsetMap.

Step 4 of 10: Offset map

An offset map records where every piece of the spoken text came from.

It is a list of spans. Each span pairs a byte range in the spoken text with a character range in the document. There are four kinds of span:

• Literal: the same text, character for character.
• Expanded: spoken words that replace a whole source token, such as "Doctor" for "Dr.". Any word inside highlights the whole token.
• Inserted: spoken words with no source, such as "heading level 2". They highlight nothing.
• Elided: source that is not spoken, such as Markdown's **. Its spoken range is empty.

The spoken text and its map are built together. They are never lined up afterwards by guessing.

Step 5 of 10: Utterance

An utterance is what goes to the speech service: the text to speak, an id, and the offset map.

The id has two parts: the reading generation and the chunk number. The generation changes every time reading stops or restarts. The chunk number counts the utterances within one reading.

Step 6 of 10: Speech service

The speech service runs on its own thread, so the screen never waits for speech.

Its queue keeps two sentences of lookahead in the engine. That way there is no gap between sentences, even with slow engines.

The service normalizes each utterance on its own, never across two utterances. The steps run in this order: math first, then Markdown residue, then abbreviations, then numbers, then punctuation. Math goes first so that a formula such as $2x$ is not read as money. Every step returns a new offset map, and the maps are chained, so the final map still points into the document.

Decided in ADR-0003: Speech threading and event timing.

Step 7 of 10: Backend or engine host

A backend turns the text into sound, either inside textweaver or in a separate host process.

Some backends run inside textweaver's own process, such as espeak-ng. Others run in a host process: ETI-Eloquence, SAPI5 voices, and DECtalk. A host loads the engine's library, speaks into memory, and sends the audio and word positions back over a framed pipe protocol. textweaver plays the audio itself. If an engine crashes, only its host stops, and textweaver restarts it.

Decided in ADR-0012: One engine-host protocol and playback client.

Step 8 of 10: Word events

Each word event gives the byte range of a spoken word and the audio time when it is heard.

For engines that report audio positions, the event is scheduled on the playback clock. So the highlight moves when you hear the word, not when the engine made it. Some engines send no word events, such as Omnivox. For them a timer pacer estimates when each word is heard.

Step 9 of 10: Generation check

Events from an older reading generation are dropped, so a late event never moves the highlight.

Suppose you press Space to pause, then jump to the next heading. The engine may still send an event for a word from before the pause. That event carries the old generation, so the service drops it before anything else sees it.

Step 10 of 10: Highlight

The word's byte range maps back through the offset map to a range in the document, and that range is highlighted.

The terminal reader draws the highlight at once. It also parks the hardware cursor on the spoken word. Screen readers and screen magnifiers follow the hardware cursor, so they follow the reading too.