<img src="/assets/img/saysynth-small.png" alt="SaySynth logo" style="width:90%;"/>
 
<img src="/assets/img/qr/slides-saysynth.png" alt="QR code to SaySynth slides" style="width:200px;"/>

Brian Abelson
 
Composition Codes, 2025-12-21
 
<a href="https://brian.abelson.live" target="_blank">brian.abelson.live</a>
 
<a href="https://gitlab.com/abelsonlive/saysynth" target="_blank">gitlab.com/abelsonlive/saysynth</a>

---

+ A brief history of speaking machines 
+ Macintalk and the `say` command
+ saysynth
+ Why does this matter? 
+ Demos

---

#### Types of Speaking Machines
<hr>

```mermaid
graph LR
 %%{init: {'theme': 'dark', 'themeVariables': { 'darkMode': true }}}%%
 Text[Input] --> Analysis[Text Analysis: Phoneme & Prosody Extraction Optional for Generative methods]
 Text --> A1
 Analysis --> F1
 Analysis --> S1
 Analysis --> N1

A2 --> Out[Audible Speech]
    F3 --> Out
    S3 --> Out
    N2 --> Out

subgraph "Mechanical"
    A1[Operator Control: Levers & Bellows set phoneme parameters]
    A1 --> A2[Sound Generation: Air pushed through reeds & modeled vocal tract / lips]
    end

subgraph "Formant/Rule-Based"
    F1[Parameter Mapping: Determining target frequencies F1-F3]
    F1 --> F2[Signal Generation: Buzz source for voiced, Noise for unvoiced]
    F2 --> F3[Digital Filtering: Resonators simulate vocal tract shape]
    end

subgraph "Sample-Based"
    S1[Sample Selection: Finding best-fit recordings in database]
    S1 --> S2[Concatenation: Splicing snippets e.g. pairs of phonemes]
    S2 --> S3[Signal Processing: Smoothing transitions & adjusting pitch]
    end

subgraph "Generative (Neural/AI)"
    N1[Acoustic Model: Predicting spectrogram parameters via NN]
    N1 --> N2[Vocoder: Predicting sample-by-sample waveforms e.g. WaveNet]
end
```

Notes:
- There are 4 main types of speaking machines illustrated here: Mechanical, Formant/Rule-Based, Sample-Based, and Generative (Neural/AI).
- Mechanical speaking machines use physical components to mimic the human vocal apparatus.
- Formant/Rule-Based synthesizers generate speech by shaping oscillators through comb filters which simulate the simulate resonant frequencies of the vocal tract.
- Sample-Based synthesizers piece together pre-recorded snippets of human speech to form new utterances.
- Generative speaking machines use neural networks to predict speech waveforms directly from text input. These are essentially sample-based synthesizers which use extremely small samples.

---

#### Von Kemplen's Speaking Machine (1773)
<hr>
<video src="/assets/video/kemplens-speaking-machine.mp4" controls style="width:80%;"></video>

Notes: 
The "first" mechanical speech synthesizer that used a bellow and levers which forced air through a reed and a series of valves to mimic human speech sounds.

---

#### Von Kemplen's Speaking Machine (1773)
<hr>
<img src="/assets/img/vonkemplen-diagram.png" style="width:80%;"/>

(Mechanical) The first mechanical speech/quack synthesizer.

Notes: 
- Special whistles to simulate noisier consonants like "s" and "sh".
- The operator used their hand to manipulate this leather tube to simulate the shape of the vocal tract and the opening of the mouth thereby producing different speech sounds.
- If you've ever heard a brass player use a mute or you've spoken with your noise closed you'll have a sense of how this works.

---

#### Joseph Faber's Euphonia (1845)
<hr>
<img src="/assets/img/euphonia.png" style="height:40vh; object-fit: contain;"/>

(Mechanical) An extremely steampunk speaking machine.

Notes:
- Basically the same mechanism as Von Kemplen's but far more sophistcated.
- There were sixteen levers or keys to trigger different phonemes.
- Oftentimes was demonstrated with a female model with a dress covering the machinery.

---

#### Edison Talking Dolls (1890s)
<hr>
<video src="/assets/video/edison-speaking-doll.mp4" controls style="width:80%;"></video>
 (Concatenative) A speaking doll powered by a phonograph.

Notes:
- These dolls contained a miniature phonograph which played back pre-recorded children's rhymes.
- Edison was partly motivated to embed this technology into a child's toy because people were initially unsettled by recorded voices.

#### VODER (1939)
<hr>
<video src="/assets/video/voder-1939.mp4" controls style="width:70%;"></video>
 (Formant-based) Pure analog speech synthesis played by "Voderettes".
---

#### VODER (1939)
<hr>
<img src="/assets/img/voder-diagram.gif" style="width:80%;"/>

---

#### Multichannel Speaking Automaton (MUSA) (1978)
<hr>
<video src="/assets/video/musa-cselt-example.mp4" controls style="width:70%;"></video>

(Concatenative) An early example of diphone speech synthesis.
Notes:
- Developed in Italy, MUSA was one of the first practical implementations of diphone synthesis, which uses recordings of the transitions between phoneme-pairs to create more natural-sounding speech.
- This technology was later commercialized in many TTS systems of the 90s and early 2000s (e.g. GPS voices, automated customer service voices, etc.)
---

#### Software Automatic Mouth (SAM) (1982)
<hr>
<img src="/assets/img/sam-ad.jpg" style="height:40vh; object-fit: contain;"/>
 
 (Formant-based) The first commercially available speech synthesizer.
<a style="font-size: 16px;" href="https://discordier.github.io/sam/" target="_blank">Online Reproduction of S.A.M.</a>

---

#### Singing Machines: The Mother of All Demos
<hr>
<img src="/assets/img/hal-9000-eye.png" style="width:50px;"/>

- Speaking machines are often demonstrated via singing.

- Speaking machines echo the biases of the cultural discourse they're created within.

Notes: 
- Across all these examples we see a couple of patterns. One is that singing is often used as a demo for speaking machines
because it highlights the system's ability to handle pitch variation, rhythm, and expressiveness, which are crucial for natural-sounding speech.
In this there is an implicit assertion that singing is the ultimate auditory expression of human language. 
That a speaking machine can't be truly human unless it can sing. The most famous example of this is HAL 9000 singing "Daisy Bell" in 2001: A Space Odyssey,
a story ostensibly of the dangers of AI surpassing human intelligence. 
- The other is of course the way in which speaking machines, like all technologies, echo the biases (esp. sexism) of the cultural discourse they're created within.
Joseph Faber choice a female face for his Euphonia most-likely because it put a less-threatening face on 
a frightening new technology. The Voder is historically celebrated as a feat of engineering but would've just been a crude monophonic synthesizer
if not for the highly skilled "Voderettes" who spent years mastering its operation. This pattern continues today as
most commercial TTS voices are codified as female

---

#### Macintalk/Fred (1984)
<hr>
<video src="/assets/video/fred-mac.mp4" controls style="width:75%;"></video>

---

#### The SAY command (2001)
<hr>
<img src="/assets/img/mac-osx-disk.png" style="width: 150px"/>

- A command-line interface to Macintalk. 
- First released in Mac OS X (Cheetah) in 2001.
- `say -v Fred I sure like being inside this fancy computer`

---

#### SAY's DSL for Prosody Control
<hr>

```
[[inpt TUNE]]
~
AA {D 120; P 176.9:0 171.4:22 161.7:61}
r {D 60; P 166.7:0}
~
y {D 210; P 161.0:0}
UW {D 70; P 178.5:0}
_
S {D 290; P 173.3:0 178.2:8 184.9:19 222.9:81}
1AX {D 280; P 234.5:0 246.1:39}
r {D 170; P 264.2:0}
~
y {D 200; P 276.9:0 274.9:17 271.0:50}
UW {D 40; P 265.0:0 264.3:50}
_
b {D 140; P 263.6:0 263.5:13 263.3:60}
r {D 110; P 263.1:0 260.4:43}
1UX {D 30; P 256.8:0 256.8:6}
S {D 190; P 256.1:0}
t {D 20; P 252.0:0 253.6:47}
~
y {D 30; P 255.5:0 257.8:45}
AO {D 40; P 260.6:0 260.0:56}
r {D 40; P 259.5:0}
_
t {D 190; P 251.3:0 250.0:16 245.9:68}
1IY {D 260; P 243.4:0 248.1:8 286.1:72 288.5:84}
T {D 220; P 291.6:0 262.8:27 220.0:67 184.8:100}
? {D 300}
[[inpt TEXT]]
```

---

#### saysynth demo
<hr>

```yaml
name: fire
globals:
    start_bpm: 65
    duration_bpm: 65
    segment_bpm: 65
    beat_bpm: 65
    note_bpm: 65
    rate: 160
    stereo: true
tracks:
    water:
        type: chord
        options:
            root: F#2
            text: wawer
            voice: Victoria
            attack: 0.3
            release: 0.7
            chord_notes: [-12, -5, 0, 4, 9, 14]
            start_count: 0
            duration_count: 112
            segment_count: 1/32
            randomize_segments: octaves,velocity
            volume_range: [0.01, 0.19]

fire:
        type: chord
        options:
            root: F#0
            chord_notes: [0, 12]
            text: fire!
            start_count: 16
            attack: 0.3
            release: 0.7
            duration_count: 96
            segment_count: 1/6
            randomize_segments: octaves,velocity
            volume_range: [0.05, 0.4]

wind:
        type: note
        options:
            root: A#3
            text: wynd!
            start_count: 24
            attack: 0.3
            release: 0.7
            duration_count: 88
            segment_count: 1/6
            randomize_segments: octaves,velocity
            volume_range: [0.00, 0.17]

sky:
        type: note
        options:
            root: C#3
            text: ohh
            start_count: 48
            attack: 0.3
            release: 0.7
            duration_count: 64
            segment_count: 1/6
            randomize_segments: octaves,velocity
            volume_range: [0.02, 0.15]

sky2:
        type: note
        options:
            root: C#3
            text: ohh
            start_count: 56
            attack: 0.3
            release: 0.7
            duration_count: 56
            segment_count: 1/6
            randomize_segments: octaves,velocity
            volume_range: [0.02, 0.15]

sky3:
        type: note
        options:
            root: G#4
            text: ohh
            start_count: 64
            attack: 0.3
            release: 0.7
            duration_count: 48
            segment_count: 1/6
            randomize_segments: octaves,velocity
            volume_range: [0.02, 0.15]
```

---

> Whatever you now find weird, ugly, uncomfortable and nasty about a new medium will surely become its signature... It’s the sound of failure: so much modern art is the sound of things going out of control, of a medium pushing to its limits and breaking apart. – Brian Eno

--- 
#### The "Singularity" According to Capitalists
<hr>
<img src="/assets/img/singularity-plot-1.png"/>

---
#### Our Lived Experience of the "Singularity"
<hr>
<img src="/assets/img/singularity-plot-2.png"/>

---
#### The Role of Art in the "Singularity"
<hr>
<img src="/assets/img/singularity-plot-2-warped.gif"/>