Troubleshooting

Solutions for common issues with PiSovereign

Quick Diagnostics

Run these commands first to identify the problem:

# Check all containers are running
docker compose ps

# Health check
curl http://localhost/health | jq

# Detailed readiness
curl http://localhost/ready/all | jq

# Recent logs
docker compose logs --tail=100 pisovereign

# System resources
docker stats --no-stream

Hailo AI HAT+

Device not detected

Symptom: Hailo device not available inside the container

Diagnosis:

# Check device files on the host
ls -la /dev/hailo*

# Check kernel module on the host
lsmod | grep hailo

# Check PCIe
lspci | grep -i hailo

Solutions:

  1. Check physical connection — ensure the HAT+ is fully seated on GPIO pins, PCIe FPC cable is connected, and you are using the 27W USB-C power supply

  2. Reinstall drivers on the host:

    sudo apt remove --purge hailo-*
sudo apt autoremove
sudo reboot
sudo apt install hailo-h10-all
sudo reboot

  3. Check device passthrough — ensure docker-compose.yml maps /dev/hailo0 into the container

Hailo firmware error

# Reset the device (on host)
sudo hailortcli fw-control reset

# Update firmware
sudo apt update && sudo apt upgrade hailo-firmware

Inference Problems

Inference timeout

Diagnosis:

# Test Ollama directly inside Docker
docker compose exec ollama curl -s http://localhost:11434/api/generate \
  -d '{"model":"qwen2.5-1.5b-instruct","prompt":"Hi","stream":false}'

Solutions:

  1. Increase timeout:

    [inference]
timeout_ms = 120000  # 2 minutes

  2. Use a smaller model:

    [inference]
default_model = "llama3.2-1b-instruct"

Model not found

# List models
docker compose exec ollama ollama list

# Pull missing model
docker compose exec ollama ollama pull qwen2.5-1.5b-instruct

Poor response quality

Adjust in config.toml:

[inference]
max_tokens = 4096
temperature = 0.5  # Lower = more focused

If model routing is enabled, ensure complex queries use a capable model:

[model_routing.models]
complex = "gemma3:12b"

Model routing — wrong tier selected

Check Prometheus metrics to see tier distribution:

curl -s http://localhost:3000/metrics/prometheus | grep model_routing

If too many requests go to the Simple tier, lower max_simple_words or add more complex_keywords:

[model_routing.classification]
max_simple_words = 10
complex_keywords = ["code", "implement", "explain", "analyze", "compare"]

Model routing — Ollama out of memory

When multiple models are loaded, Ollama may run out of RAM. Reduce the number of concurrent models:

# compose.yml
OLLAMA_MAX_LOADED_MODELS: 1

Or use smaller models for the Simple and Moderate tiers.

Network & Connectivity

Connection refused

Diagnosis:

# Check containers
docker compose ps

# Check Traefik is routing
docker compose logs traefik | tail -20

# Test direct container access
docker compose exec pisovereign curl -s http://localhost:3000/health

Solutions:

  1. Check bind address in config.toml:

    [server]
host = "0.0.0.0"

  2. Check Traefik configuration — verify domain and routing rules in docker/traefik/dynamic.yml

TLS/SSL errors

  • Development: Use http://localhost (Traefik handles TLS for external access)
  • Production: Ensure your domain’s DNS points to the server, and Let’s Encrypt can reach port 80 for validation
  • Self-signed certs (e.g., Proton Bridge): set verify_certificates = false in the relevant config section

Database Issues

Database locked

Cause: Multiple concurrent writers to SQLite

Solutions:

  1. Ensure single PiSovereign instance:

    docker compose ps | grep pisovereign
# Should show exactly one instance

  2. Verify WAL mode:

    docker compose exec pisovereign sqlite3 /data/pisovereign.db "PRAGMA journal_mode;"
# Should return "wal"

Migration failed

# Backup current database
docker compose exec pisovereign cp /data/pisovereign.db /data/pisovereign-backup.db

# Reset database (LOSES DATA — restore from backup afterward)
docker compose exec pisovereign rm /data/pisovereign.db
docker compose restart pisovereign

Database corruption

# Attempt recovery
docker compose exec pisovereign sh -c \
  'sqlite3 /data/pisovereign.db ".recover" | sqlite3 /data/pisovereign-recovered.db'

Integration Problems

WhatsApp

Webhook verification failed:

  1. URL must be publicly accessible — test with curl from an external network
  2. verify_token in config.toml must match the Meta developer console
  3. HTTPS must be configured (Traefik handles this)

Messages not received:

  1. Check webhook is subscribed to the messages field in Meta console
  2. Verify phone number is whitelisted (for test numbers)
  3. Check logs: docker compose logs pisovereign | grep -i whatsapp

Email (IMAP/SMTP)

Connection refused:

# Test IMAP from the container
docker compose exec pisovereign openssl s_client -connect imap.gmail.com:993
  • Verify host/port match your provider (see External Services)
  • For Proton Bridge: ensure Bridge is running on the host
  • If using the provider field, explicit imap_host/smtp_host values override presets

Authentication failed:

  • Gmail: Use an App Password, not your account password
  • Outlook: Use an App Password if 2FA is enabled
  • Proton Mail: Use the Bridge Password from the Bridge UI, not your Proton account password

Migrating from [proton] config: The old [proton] config section still works via a serde alias. If you see “duplicate field” errors, ensure you don’t have both [proton] and [email] sections in your config.

CalDAV

401 Unauthorized:

docker compose exec pisovereign curl -u username:password \
  http://baikal:80/dav.php/calendars/username/

Verify user exists in Baïkal admin at http://localhost/caldav.

404 Not Found — PiSovereign automatically re-creates missing calendars and address books. If you still see 404 errors:

  • Verify calendar_path matches your Baïkal user and calendar name
  • Check that the user has permissions to create calendars
  • List calendars to verify:
docker compose exec pisovereign curl -u username:password -X PROPFIND \
  http://baikal:80/dav.php/calendars/username/

Speech Processing

Whisper (STT) fails

# Check Whisper container
docker compose logs whisper

# Test directly
docker compose exec whisper curl -s http://localhost:8081/health
  • Verify the container has enough memory (~500 MB for base model)
  • Check audio format (mono 16 kHz WAV preferred)

Piper (TTS) fails

# Check Piper container
docker compose logs piper

# Test directly
docker compose exec piper curl -s http://localhost:8082/health
  • Verify voice model files are mounted correctly
  • Check container logs for ONNX runtime errors

Memory System (RAG)

Memories not being retrieved

  1. Check that enable_rag = true in [memory] config
  2. Verify rag_threshold isn’t too high — try lowering to 0.3
  3. Ensure embeddings are generated: GET /v1/memories/stats should show entries with embeddings
  4. Confirm Ollama is running with the nomic-embed-text embedding model

Encryption key errors

  • “Read-only file system”: Ensure encryption_key_path points to a writable directory (e.g., /app/data/memory_encryption.key in Docker)
  • Lost encryption key: Encrypted memories cannot be recovered. Delete the key file, clear the memories and memory_embeddings tables, and let PiSovereign generate a new key on startup

Memory decay not running

The decay task runs automatically every 24 hours. Check logs for memory decay task entries. You can also trigger it manually via POST /v1/memories/decay.

System Commands

Commands not auto-populating

On first startup, 32 default system commands should be auto-discovered. If empty:

  1. Check logs for system_command_discovery entries
  2. Verify the database migration 14_system_commands.sql ran successfully
  3. Check GET /v1/commands/catalog/count — should return {"count": 32}

Startup warnings about Vault credentials

PiSovereign validates Vault credentials at startup and logs diagnostic warnings for missing or empty fields. Check the log line Some configuration fields are empty after secret resolution for affected fields. Store the missing secrets in Vault using just docker-vault-check.

Performance Issues

High memory usage

docker stats --no-stream

Reduce resource consumption in config.toml:

[cache]
l1_max_entries = 1000

[database]
max_connections = 3

Slow response times

  1. Check inference latency — the model may be too large for your hardware
  2. Enable caching: [cache] enabled = true
  3. Use SSD storage — SD card I/O is a common bottleneck on Raspberry Pi

Getting Help

Collect Diagnostic Information

Before reporting an issue, gather:

# Container status
docker compose ps

# PiSovereign version
docker compose exec pisovereign pisovereign-server --version

# Recent logs
docker compose logs --since "1h" > pisovereign-logs.txt

# System info
uname -a
docker --version
free -h
df -h

Report an Issue