Voice Integration Specification

Version: 0.1 (Draft)
Date: March 2026
Status: Proposal
Depends on: Tutor Behavior Spec
Used by: Tutor

1. Overview

Voice is how the Tutor speaks and listens. It must feel natural — like talking to a real person, not a voice assistant.

The voice pipeline runs on the Tutor device — which can be a phone, laptop, or browser tab. The tablet (Edge) has no audio role.

1.1 Deployment Options

Recommendation: Browser-first for v1, with PWA wrapper for "installed" feel.

1.2 Design Goals

1.3 Non-Goals

• Wake word ("Hey Tutor") — session is explicit, always listening
• Multi-speaker recognition — only student speaks to Tutor
• Background music/audio — educational context only

2. Architecture

2.1 Pipeline Overview

┌─────────────────────────────────────────────────────────────────┐
│                       VOICE PIPELINE                            │
│                                                                 │
│  ┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────┐     │
│  │  Audio  │    │   VAD   │    │   STT   │    │  Tutor  │     │
│  │ Capture │───►│         │───►│         │───►│  Core   │     │
│  │         │    │ (Voice  │    │ (Speech │    │         │     │
│  │         │    │  Detect)│    │ to Text)│    │         │     │
│  └─────────┘    └─────────┘    └─────────┘    └────┬────┘     │
│                                                     │          │
│                                                     ▼          │
│  ┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────┐     │
│  │  Audio  │    │   Mix   │    │   TTS   │    │ Response│     │
│  │ Output  │◄───│         │◄───│         │◄───│  Gen    │     │
│  │         │    │         │    │ (Text   │    │         │     │
│  │         │    │         │    │ to Spch)│    │         │     │
│  └─────────┘    └─────────┘    └─────────┘    └─────────┘     │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

2.2 Component Responsibilities

3. Speech-to-Text (STT)

3.1 Requirements

3.2 Provider Options

Recommendation: Whisper (local) for privacy and cost. Fall back to Deepgram for poor local performance.

3.3 Streaming STT

STT provides partial results while student speaks:

Time    Audio                    Partial Results
────    ─────                    ───────────────
0.0s    "What..."               
0.3s                             "What"
0.5s    "...should..."          
0.8s                             "What should"
1.0s    "...I do..."            
1.3s                             "What should I do"
1.5s    "...first?"             
2.0s    (silence detected)       
2.2s                             "What should I do first?" [FINAL]

3.4 Math Vocabulary

Custom vocabulary boost for:

Numbers: zero through twenty, hundred, thousand
Variables: x, y, z, n, a, b, c
Operations: plus, minus, times, divided by, equals, squared, cubed
Terms: equation, expression, fraction, numerator, denominator
       exponent, coefficient, variable, constant, term
       positive, negative, both sides, isolate, solve

3.5 Handling Math Speech

4. Text-to-Speech (TTS)

4.1 Requirements

4.2 Provider Options

Recommendation: ElevenLabs for quality and emotion. Voice cloning for consistent persona.

4.3 Voice Configuration

{
  "voice_id": "freaking_genius_tutor_v1",
  "model": "eleven_turbo_v2",
  "settings": {
    "stability": 0.65,
    "similarity_boost": 0.75,
    "style": 0.35,
    "use_speaker_boost": true
  },
  "generation_config": {
    "optimize_streaming_latency": 3,
    "output_format": "mp3_44100_128"
  }
}

4.4 Prosody Control

Tune speech based on context:

4.5 Speech Synthesis Markup (SSML)

For nuanced control:

<speak>
  <prosody rate="95%">
    What happens <break time="300ms"/> 
    when you move the five 
    <emphasis level="moderate">to the other side</emphasis>?
  </prosody>
</speak>

5. Voice Activity Detection (VAD)

5.1 VAD Behavior

                    SILENCE          SPEECH          SILENCE
Audio:     ─────────────────│███████████████│─────────────────
                            │               │
                            ▼               ▼
State:     NOT_SPEAKING → SPEAKING → TRAILING → NOT_SPEAKING
                            │               │
Events:         speech_start         speech_end (after 1s silence)

5.2 VAD Parameters

5.3 Noise Handling

• Apply noise suppression before VAD
• Calibrate to ambient noise level on session start
• Adapt threshold during session

6. Turn-Taking

6.1 Turn States

┌─────────────────────────────────────────────────────────────────┐
│                       TURN-TAKING                               │
│                                                                 │
│   ┌───────────┐     Student speaks      ┌───────────┐         │
│   │  TUTOR    │ ───────────────────────►│  STUDENT  │         │
│   │  TURN     │                          │  TURN     │         │
│   │           │◄─────────────────────────│           │         │
│   │ (speaking │     Tutor responds       │ (speaking │         │
│   │  or       │                          │  or       │         │
│   │  silent)  │◄─────────────────────────│  writing) │         │
│   └───────────┘     Student finishes,    └───────────┘         │
│                     Tutor decides to                            │
│                     speak                                       │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

6.2 Interruption Handling

When student speaks while Tutor is speaking:

Tutor speaking: "When you move the five to the other side—"
                                                    │
Student: "Wait, what?"                              │
                                                    │
1. TTS stops immediately ◄──────────────────────────┘
2. STT processes student speech
3. Tutor responds to interruption

Tutor: "Go ahead, what's your question?"

6.3 Barge-In Sensitivity

Detect backchannel vs. interruption using:

• Duration (backchannels are short)
• Prosody (backchannels have characteristic pattern)
• Keywords ("wait", "hold on", "but" = interrupt)

7. Audio Processing

7.1 Input Processing

Microphone → Noise Suppression → AGC → VAD → STT
                    │
                    └── Echo Cancellation (if Tutor playing)

7.2 Output Processing

TTS → Volume Normalization → Ducking (if student speaks) → Speaker

7.3 Audio Format

8. Latency Budget

8.1 End-to-End Target

Student finishes speaking → Tutor starts responding: <800ms

8.2 Budget Breakdown

8.3 Perceived Latency

Even if processing takes time, Tutor can:

• Use filler: "Hmm..." "Let's see..." while thinking
• Start TTS with common phrase while generating rest

9. Language Support

9.1 Initial Languages

9.2 Language Detection

• Set per-student profile (not auto-detected)
• Support code-switching within session (future)

9.3 Accent Handling

Train/fine-tune STT for:

• Children's voices (higher pitch, less clear articulation)
• Regional accents (Israeli English, British variations)
• Math-specific pronunciations

10. Offline Capability

10.1 Offline Mode

When internet unavailable:

10.2 Fallback Behavior

1. Detect connectivity loss
2. Switch to offline models
3. Notify student: "I'm having trouble connecting. I'll do my best."
4. Resume cloud services when available

11. Privacy & Security

11.1 Audio Data Handling

11.2 Privacy Modes

11.3 Compliance

• COPPA compliant (parental consent for minors)
• GDPR compliant (data minimization, deletion rights)
• Audio encrypted in transit (TLS)

12. Platform Integration

12.1 Browser (Web)

Audio capture:

// Request microphone access
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const audioContext = new AudioContext();
const source = audioContext.createMediaStreamSource(stream);

Constraints:

Browser support:

12.2 Phone (Native)

iOS:

let audioSession = AVAudioSession.sharedInstance()
try audioSession.setCategory(.playAndRecord, mode: .voiceChat)
try audioSession.setActive(true)

Android:

val audioManager = getSystemService(Context.AUDIO_SERVICE) as AudioManager
audioManager.mode = AudioManager.MODE_IN_COMMUNICATION

Interruption handling:

12.3 Desktop (Native/Electron)

Advantages:

• Best audio latency
• No permission prompts after initial grant
• Background operation
• System audio integration

Considerations:

• Electron/Tauri for cross-platform
• Native builds for best performance

12.4 Hardware Requirements

12.5 Audio Session Management

When Tutor session is active:

1. Claim audio focus — pause other audio sources
2. Configure for voice — optimize for speech, not music
3. Handle interruptions — pause gracefully, resume cleanly
4. Manage permissions — request once, remember grant

13. Testing

13.1 STT Testing

13.2 TTS Testing

13.3 Integration Testing

14. Metrics

14.1 Quality Metrics

14.2 Usage Metrics

Appendix A: Provider Integration

ElevenLabs Setup

from elevenlabs import generate, stream

def speak(text):
    audio_stream = generate(
        text=text,
        voice="freaking_genius_tutor_v1",
        model="eleven_turbo_v2",
        stream=True
    )
    stream(audio_stream)

Whisper Local Setup

import whisper

model = whisper.load_model("base.en")  # or "small.en" for better quality

def transcribe(audio_path):
    result = model.transcribe(audio_path)
    return result["text"]

Deepgram Streaming

from deepgram import Deepgram

dg = Deepgram(API_KEY)

async def transcribe_stream(audio_stream):
    socket = await dg.transcription.live({
        "model": "nova-2",
        "language": "en-US",
        "smart_format": True
    })
    
    socket.on("transcript", handle_transcript)
    # ... stream audio to socket

Appendix B: Troubleshooting

This completes Phase 2 specs. Ready for implementation.