Voice-First Interface

🎙️ Multi-room voice interaction with wake word detection, speaker identification, and privacy controls

This document covers the architecture, data flow, configuration, and deployment of PiSovereign’s voice-first interface.

Table of Contents

• Overview
• Architecture
  • Component Diagram
  • Audio Pipeline Flow
  • Voice Session State Machine
• Domain Model
  • Voice Room
  • Voice Session
  • Voice Profile & Speaker Identification
• Application Services
  • VoiceRoomService
  • VoiceSessionService
  • SpeakerEnrollmentService
  • VoicePipelineService
• Port Traits
• MQTT Topic Structure
• Docker Deployment
  • Services
  • Starting the Voice Stack
  • Resource Requirements
• Configuration Reference
• REST API Endpoints
• Privacy & Whisper Mode
• Troubleshooting

Overview

The voice-first interface turns PiSovereign into a hands-free AI assistant. Audio streams from satellite devices (Raspberry Pi Zero 2W, ESP32-S3, etc.) over MQTT. The server handles wake word detection, speech-to-text, LLM inference, text-to-speech, and speaker identification — all locally, with no cloud dependency.

Key capabilities:

• Multi-room audio — Independent rooms with individual volume, mute, and online status via MQTT
• Wake word detection — “Sovereign” (customizable) via openWakeWord with configurable sensitivity
• Continuous conversation — Follow-up window after a response, no re-triggering needed
• Speaker identification — ECAPA-TDNN embeddings match voice to enrolled user profiles
• Whisper mode — Automatic volume reduction during quiet hours
• Privacy LED — GPIO-controlled indicator when microphone is active (Raspberry Pi only)

Architecture

Component Diagram

graph TB
    subgraph Satellites["Room Satellites"]
        SAT1["🎙️ Kitchen<br/>RPi Zero 2W"]
        SAT2["🎙️ Office<br/>ESP32-S3"]
        SAT3["🎙️ Bedroom<br/>RPi Zero 2W"]
    end

    subgraph Docker["Docker Stack (voice profile)"]
        MQTT["Mosquitto<br/>MQTT Broker"]
        OWW["openWakeWord<br/>Wake Word Detection"]
        SID["Speaker-ID<br/>ECAPA-TDNN"]
        APP["PiSovereign<br/>Voice Pipeline"]
        STT["Whisper<br/>Speech-to-Text"]
        TTS["Piper<br/>Text-to-Speech"]
        LLM["Ollama<br/>LLM Inference"]
    end

    SAT1 & SAT2 & SAT3 -->|"Audio PCM<br/>QoS 0"| MQTT
    MQTT -->|"Subscribe"| APP
    APP -->|"Audio chunks"| OWW
    APP -->|"Voice sample"| SID
    APP -->|"Speech audio"| STT
    APP -->|"Response text"| TTS
    APP -->|"User query"| LLM
    APP -->|"TTS audio<br/>QoS 1"| MQTT
    MQTT -->|"Playback"| SAT1 & SAT2 & SAT3

Audio Pipeline Flow

sequenceDiagram
    participant Sat as Room Satellite
    participant MQTT as Mosquitto
    participant VP as Voice Pipeline
    participant WW as openWakeWord
    participant STT as Whisper STT
    participant LLM as Ollama
    participant TTS as Piper TTS
    participant SID as Speaker-ID

    Sat->>MQTT: Publish audio (PCM 16-bit, 16kHz)
    MQTT->>VP: Deliver audio chunk
    VP->>WW: Check for wake word
    WW-->>VP: Detection (word, confidence)

    Note over VP: Wake word detected → start session

    VP->>SID: Identify speaker (audio sample)
    SID-->>VP: SpeakerMatch {id, confidence}
    VP->>VP: Activate privacy LED (GPIO)
    VP->>STT: Transcribe speech
    STT-->>VP: Transcription text
    VP->>LLM: Generate response
    LLM-->>VP: Response text
    VP->>TTS: Synthesize speech
    TTS-->>VP: Audio PCM
    VP->>MQTT: Publish response audio
    MQTT->>Sat: Play response

    Note over VP: Enter follow-up window (10s default)

Voice Session State Machine

stateDiagram-v2
    [*] --> Listening: Wake word detected
    Listening --> Processing: Speech captured
    Processing --> Responding: LLM response ready
    Responding --> FollowUp: TTS playback complete
    FollowUp --> Listening: Follow-up speech detected
    FollowUp --> Ended: Timeout (10s default)
    Listening --> Ended: Max duration (5 min)
    Processing --> Ended: Error / timeout
    Responding --> Ended: Error

Domain Model

Voice Room

A VoiceRoom represents a physical location with a satellite audio device.

Rooms go offline automatically when heartbeats stop arriving (configurable timeout, default 30s).

Voice Session

A VoiceSession tracks a single voice interaction within a room.

Voice Profile & Speaker Identification

A VoiceProfile stores enrolled speaker embeddings for recognition.

• Embedding model: ECAPA-TDNN (SpeechBrain), 192-dimensional vectors
• Minimum enrollment: 3 audio samples for reliable identification
• Matching: Cosine similarity between embeddings, configurable threshold (default 0.75)
• Audio format: PCM 16-bit little-endian, 16 kHz, mono

graph LR
    subgraph Enrollment["Speaker Enrollment (3+ samples)"]
        A1["🎤 Sample 1"] --> E1["Embedding"]
        A2["🎤 Sample 2"] --> E2["Embedding"]
        A3["🎤 Sample 3"] --> E3["Embedding"]
    end

    subgraph Recognition["Runtime Recognition"]
        AX["🎤 Live Audio"] --> EX["Live Embedding"]
        EX --> COS["Cosine Similarity"]
        E1 & E2 & E3 --> COS
        COS -->|"> 0.75"| MATCH["✅ Speaker Identified"]
        COS -->|"≤ 0.75"| UNKNOWN["❓ Unknown Speaker"]
    end

Application Services

VoiceRoomService

Manages room lifecycle and health monitoring.

A background task calls check_stale_rooms() every 30 seconds.

VoiceSessionService

Manages voice session state transitions.

SpeakerEnrollmentService

Manages speaker profile enrollment and identification data.

VoicePipelineService

Orchestrates the full voice interaction pipeline (wake word → STT → LLM → TTS).

Port Traits

The voice subsystem defines 7 port traits in crates/application/src/ports/:

All ports use #[async_trait] and support mockall via #[cfg_attr(test, automock)].

MQTT Topic Structure

All topics use a configurable prefix (default: pisovereign).

Audio format: PCM 16-bit little-endian, 16 kHz, mono.

Docker Deployment

Services

The voice stack consists of three services, activated via the voice Docker Compose profile:

All services run on the internal pisovereign-network and are never exposed externally.

Starting the Voice Stack

# Start core + voice services
just docker-up  # if voice profile is in COMPOSE_PROFILES

# Or explicitly with the voice profile
docker compose --profile voice up -d

# Verify services are healthy
docker compose --profile voice ps

Resource Requirements

The speaker-id service downloads ECAPA-TDNN models (~80 MB) on first start. Models are persisted in the speaker-id-models Docker volume.

Configuration Reference

Add to config.toml to enable the voice interface:

[voice]
enabled = true

[voice.mqtt]
broker_url = "mqtt://mosquitto:1883"
client_id = "pisovereign-voice"
keep_alive_secs = 30
max_inflight = 100
topic_prefix = "pisovereign"

[voice.wake_word]
service_url = "http://openwakeword:8083"
words = ["sovereign"]
sensitivity = 0.5          # 0.0–1.0, higher = fewer false positives
timeout_ms = 2000

[voice.speaker_id]
service_url = "http://speaker-id:8084"
min_enrollment_samples = 3
match_threshold = 0.75     # Cosine similarity threshold
timeout_ms = 5000

[voice.conversation]
follow_up_window_ms = 10000    # 10s follow-up after response
max_session_duration_ms = 300000  # 5 minutes max

[voice.whisper_mode]
enabled = true
quiet_start = "22:00"
quiet_end = "07:00"
quiet_volume = 30          # 0–100

[voice.gpio]
enabled = false            # Only on Raspberry Pi (aarch64 Linux)
privacy_led_pin = 17       # BCM GPIO pin number

[voice.rooms]
default_volume = 80        # 0–100
heartbeat_timeout_ms = 30000  # Mark offline after 30s silence

All values shown are defaults. The [voice] section is optional — when omitted, the voice subsystem is disabled.

REST API Endpoints

All voice endpoints require API key authentication.

Room Management

Speaker Management

Status

Example: Register a Room

curl -X POST http://localhost:3000/v1/voice/rooms \
  -H "Authorization: Bearer sk-your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"name": "Kitchen"}'

Response:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "name": "Kitchen",
  "is_online": false,
  "volume": 80,
  "muted": false,
  "last_seen": "2025-01-15T10:30:00Z",
  "created_at": "2025-01-15T10:30:00Z"
}

Example: Voice Status

curl http://localhost:3000/v1/voice/status \
  -H "Authorization: Bearer sk-your-api-key"

Response (voice disabled):

{
  "enabled": false,
  "wake_word_available": false,
  "speaker_id_available": false,
  "mqtt_connected": false,
  "active_rooms": 0,
  "active_sessions": 0
}

Privacy & Whisper Mode

Privacy LED

On Raspberry Pi, a GPIO-connected LED indicates when the microphone is active:

• LED on: Audio is being captured and processed
• LED off: No active voice session

Configure the BCM GPIO pin in [voice.gpio]. Requires aarch64 Linux — the feature compiles to a no-op on other platforms.

Whisper Mode

During quiet hours (default 22:00–07:00), whisper mode automatically:

1. Reduces TTS playback volume to the configured level (default 30%)
2. Uses softer TTS voice parameters when available
3. Restores normal volume outside quiet hours

Troubleshooting

Voice services not starting

# Check if voice profile is enabled
docker compose --profile voice ps

# Check individual service logs
docker compose --profile voice logs mosquitto
docker compose --profile voice logs openwakeword
docker compose --profile voice logs speaker-id

Wake word not detected

• Verify openWakeWord is healthy: curl http://localhost:8083/health
• Increase sensitivity (closer to 1.0) in [voice.wake_word]
• Check audio format: must be PCM 16-bit LE, 16 kHz, mono
• Check MQTT connectivity: mosquitto_sub -t 'pisovereign/audio/#' -v

Speaker not recognized

• Ensure at least 3 enrollment samples are recorded
• Lower match_threshold (default 0.75) if false negatives are high
• Re-enroll in a quiet environment for better embeddings
• Check speaker-id service health: curl http://localhost:8084/health

Room shows offline

• Check satellite heartbeat interval (must be < heartbeat_timeout_ms)
• Verify MQTT topic: pisovereign/status/{room_id}
• Check Mosquitto logs for connection issues

GPIO LED not working

• Only supported on Raspberry Pi (aarch64 Linux)
• Verify BCM pin number matches physical wiring
• Check GPIO permissions (user must be in gpio group or run as root)