Backup & Restore

💾 Protect your PiSovereign data with comprehensive backup strategies

This guide covers backup procedures, automated backups, and disaster recovery.

Table of Contents


Overview

Backup strategy overview:

ComponentMethodFrequencyRetention
Databasepg_dumpDaily7 daily, 4 weekly, 12 monthly
ConfigurationFile copyOn change5 versions
Vault SecretsVault backupWeekly4 weekly
Full SystemSD/NVMe imageMonthly3 monthly

What to Back Up

Critical Data

PathContentsPriority
PostgreSQL database (via pg_dump)Conversations, approvals, audit logsHigh
/etc/pisovereign/config.tomlApplication configurationHigh
/opt/vault/dataVault storage (if local)High

Important Data

PathContentsPriority
/var/lib/pisovereign/cache.redbPersistent cacheMedium
/opt/hailo/modelsDownloaded modelsMedium
/etc/pisovereign/envEnvironment overridesMedium

Can Be Recreated

PathContentsPriority
Prometheus dataMetricsLow
Grafana dashboardsCan reimportLow
Log filesHistorical onlyLow

Database Backup

Manual Backup

Using the PiSovereign CLI:

# Simple local backup
pisovereign-cli backup \
  --database-url postgres://pisovereign:pisovereign@postgres:5432/pisovereign \
  --output /backup/pisovereign-$(date +%Y%m%d).sql

# With timestamp
pisovereign-cli backup \
  --database-url postgres://pisovereign:pisovereign@postgres:5432/pisovereign \
  --output /backup/pisovereign-$(date +%Y%m%d_%H%M%S).sql

# Compressed backup
pisovereign-cli backup \
  --database-url postgres://pisovereign:pisovereign@postgres:5432/pisovereign \
  --output - | gzip > /backup/pisovereign-$(date +%Y%m%d).sql.gz

Using pg_dump directly:

# Custom format backup (most flexible, supports parallel restore)
pg_dump -Fc -h postgres -U pisovereign -d pisovereign \
  -f /backup/pisovereign-$(date +%Y%m%d).dump

# Plain SQL backup
pg_dump -h postgres -U pisovereign -d pisovereign \
  -f /backup/pisovereign-$(date +%Y%m%d).sql

Automated Backups

Create backup script:

sudo nano /usr/local/bin/pisovereign-backup.sh
#!/bin/bash
set -euo pipefail

# Configuration
BACKUP_DIR="/backup/pisovereign"
DB_URL="postgres://pisovereign:pisovereign@postgres:5432/pisovereign"
RETENTION_DAILY=7
RETENTION_WEEKLY=4
RETENTION_MONTHLY=12

# Create directories
mkdir -p "$BACKUP_DIR"/{daily,weekly,monthly}

# Timestamp
DATE=$(date +%Y%m%d)
DAY_OF_WEEK=$(date +%u)
DAY_OF_MONTH=$(date +%d)

# Daily backup (custom format for flexible restore)
DAILY_FILE="$BACKUP_DIR/daily/pisovereign-$DATE.dump.gz"
echo "Creating daily backup: $DAILY_FILE"
pg_dump -Fc -d "$DB_URL" | gzip > "$DAILY_FILE"

# Weekly backup (Sunday)
if [ "$DAY_OF_WEEK" -eq 7 ]; then
    WEEKLY_FILE="$BACKUP_DIR/weekly/pisovereign-week$(date +%V)-$DATE.dump.gz"
    echo "Creating weekly backup: $WEEKLY_FILE"
    cp "$DAILY_FILE" "$WEEKLY_FILE"
fi

# Monthly backup (1st of month)
if [ "$DAY_OF_MONTH" -eq "01" ]; then
    MONTHLY_FILE="$BACKUP_DIR/monthly/pisovereign-$(date +%Y%m).dump.gz"
    echo "Creating monthly backup: $MONTHLY_FILE"
    cp "$DAILY_FILE" "$MONTHLY_FILE"
fi

# Cleanup old backups
echo "Cleaning up old backups..."
find "$BACKUP_DIR/daily" -name "*.dump.gz" -mtime +$RETENTION_DAILY -delete
find "$BACKUP_DIR/weekly" -name "*.dump.gz" -mtime +$((RETENTION_WEEKLY * 7)) -delete
find "$BACKUP_DIR/monthly" -name "*.dump.gz" -mtime +$((RETENTION_MONTHLY * 30)) -delete

# Backup config
CONFIG_BACKUP="$BACKUP_DIR/config/config-$DATE.toml"
mkdir -p "$BACKUP_DIR/config"
cp /etc/pisovereign/config.toml "$CONFIG_BACKUP"
find "$BACKUP_DIR/config" -name "*.toml" -mtime +30 -delete

echo "Backup completed successfully"
sudo chmod +x /usr/local/bin/pisovereign-backup.sh

Schedule with cron:

sudo crontab -e
# Daily backup at 2 AM
0 2 * * * /usr/local/bin/pisovereign-backup.sh >> /var/log/pisovereign-backup.log 2>&1

S3-Compatible Storage

S3 Configuration

PiSovereign CLI supports S3-compatible storage (AWS S3, MinIO, Backblaze B2):

# Environment variables
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"

Or in configuration file:

# /etc/pisovereign/backup.toml
[s3]
bucket = "pisovereign-backups"
region = "eu-central-1"
endpoint = "https://s3.eu-central-1.amazonaws.com"
# For MinIO or Backblaze B2:
# endpoint = "https://s3.example.com"

S3 Backup Commands

# Backup to S3
pisovereign-cli backup \
  --s3-bucket pisovereign-backups \
  --s3-region eu-central-1 \
  --s3-prefix daily/ \
  --s3-access-key "$AWS_ACCESS_KEY_ID" \
  --s3-secret-key "$AWS_SECRET_ACCESS_KEY"

# With custom endpoint (MinIO)
pisovereign-cli backup \
  --s3-bucket pisovereign-backups \
  --s3-endpoint https://minio.local:9000 \
  --s3-access-key "$MINIO_ACCESS_KEY" \
  --s3-secret-key "$MINIO_SECRET_KEY"

# List backups in S3
aws s3 ls s3://pisovereign-backups/daily/

Automated S3 backup script:

#!/bin/bash
set -euo pipefail

DATE=$(date +%Y%m%d)

# Upload to S3
pisovereign-cli backup \
  --s3-bucket pisovereign-backups \
  --s3-region eu-central-1 \
  --s3-prefix "daily/pisovereign-$DATE.dump.gz" \
  --s3-access-key "$AWS_ACCESS_KEY_ID" \
  --s3-secret-key "$AWS_SECRET_ACCESS_KEY"

# Configure S3 lifecycle for automatic cleanup (one-time setup)
# aws s3api put-bucket-lifecycle-configuration \
#   --bucket pisovereign-backups \
#   --lifecycle-configuration file://lifecycle.json

S3 lifecycle policy (lifecycle.json):

{
  "Rules": [
    {
      "ID": "DeleteOldDailyBackups",
      "Status": "Enabled",
      "Filter": { "Prefix": "daily/" },
      "Expiration": { "Days": 7 }
    },
    {
      "ID": "DeleteOldWeeklyBackups",
      "Status": "Enabled",
      "Filter": { "Prefix": "weekly/" },
      "Expiration": { "Days": 30 }
    },
    {
      "ID": "DeleteOldMonthlyBackups",
      "Status": "Enabled",
      "Filter": { "Prefix": "monthly/" },
      "Expiration": { "Days": 365 }
    }
  ]
}

Full System Backup

SD Card / NVMe Image

Create full system image for disaster recovery:

# Identify storage device
lsblk

# Create image (run from another system or boot USB)
sudo dd if=/dev/mmcblk0 of=/backup/pisovereign-full-$(date +%Y%m%d).img bs=4M status=progress

# Compress (takes a while)
gzip /backup/pisovereign-full-$(date +%Y%m%d).img

Incremental System Backup

Using rsync for incremental backups:

#!/bin/bash
# /usr/local/bin/pisovereign-system-backup.sh

BACKUP_DIR="/backup/system"
DATE=$(date +%Y%m%d)
LATEST="$BACKUP_DIR/latest"

mkdir -p "$BACKUP_DIR/$DATE"

rsync -aHAX --delete \
  --exclude='/proc/*' \
  --exclude='/sys/*' \
  --exclude='/dev/*' \
  --exclude='/tmp/*' \
  --exclude='/run/*' \
  --exclude='/mnt/*' \
  --exclude='/media/*' \
  --exclude='/backup/*' \
  --link-dest="$LATEST" \
  / "$BACKUP_DIR/$DATE/"

rm -f "$LATEST"
ln -s "$BACKUP_DIR/$DATE" "$LATEST"

Restore Procedures

Database Restore

# Stop the service
sudo systemctl stop pisovereign

# Create a backup of the current database (just in case)
pg_dump -Fc -h postgres -U pisovereign -d pisovereign \
  -f /tmp/pisovereign-pre-restore.dump

# Restore from backup (custom format)
gunzip -c /backup/pisovereign/daily/pisovereign-20260207.dump.gz > /tmp/restore.dump
pg_restore -h postgres -U pisovereign -d pisovereign --clean --if-exists /tmp/restore.dump
rm /tmp/restore.dump

# Or using CLI
pisovereign-cli restore \
  --database-url postgres://pisovereign:pisovereign@postgres:5432/pisovereign \
  --input /backup/pisovereign-20260207.dump

# Verify database connectivity and integrity
pg_isready -h postgres -U pisovereign -d pisovereign
psql -h postgres -U pisovereign -d pisovereign -c "SELECT 1;"

# Start service
sudo systemctl start pisovereign

# Verify
pisovereign-cli status

Restore from S3

# Download from S3
aws s3 cp s3://pisovereign-backups/daily/pisovereign-20260207.dump.gz /tmp/

# Or using CLI
pisovereign-cli restore \
  --s3-bucket pisovereign-backups \
  --s3-key daily/pisovereign-20260207.dump.gz \
  --s3-region eu-central-1

Configuration Restore

# Restore config
sudo cp /backup/pisovereign/config/config-20260207.toml /etc/pisovereign/config.toml

# Verify syntax
pisovereign-cli config validate

# Restart service
sudo systemctl restart pisovereign

Disaster Recovery

Complete system recovery procedure:

  1. Flash fresh Raspberry Pi OS
# On another computer, flash SD card
# Use Raspberry Pi Imager
  1. Basic system setup
# SSH in, update system
sudo apt update && sudo apt upgrade -y
  1. Restore from full image (if available)
# On another system
gunzip -c pisovereign-full-20260207.img.gz | sudo dd of=/dev/mmcblk0 bs=4M status=progress
  1. Or restore components
# Install PiSovereign
# (Follow installation guide)

# Restore configuration
sudo mkdir -p /etc/pisovereign
sudo cp config.toml.backup /etc/pisovereign/config.toml

# Restore database
pisovereign-cli restore \
  --database-url postgres://pisovereign:pisovereign@postgres:5432/pisovereign \
  --input pisovereign-backup.dump

# Restore Vault (if using local Vault)
sudo tar -xzf vault-backup.tar.gz -C /opt/vault/

# Start services
sudo systemctl start pisovereign

Backup Verification

Verify Database Backup

# Check file integrity
gzip -t /backup/pisovereign/daily/pisovereign-20260207.dump.gz && echo "OK"

# Test restore to a temporary database
createdb -h postgres -U pisovereign pisovereign_verify
gunzip -c /backup/pisovereign/daily/pisovereign-20260207.dump.gz | \
  pg_restore -h postgres -U pisovereign -d pisovereign_verify
psql -h postgres -U pisovereign -d pisovereign_verify \
  -c "SELECT COUNT(*) FROM conversations;"
dropdb -h postgres -U pisovereign pisovereign_verify

Automated Verification

#!/bin/bash
# /usr/local/bin/verify-backup.sh

DB_URL="postgres://pisovereign:pisovereign@postgres:5432"
BACKUP_FILE="/backup/pisovereign/daily/pisovereign-$(date +%Y%m%d).dump.gz"

if [ ! -f "$BACKUP_FILE" ]; then
    echo "ERROR: Today's backup not found!"
    exit 1
fi

# Verify gzip integrity
if ! gzip -t "$BACKUP_FILE" 2>/dev/null; then
    echo "ERROR: Backup file is corrupted!"
    exit 1
fi

# Verify database integrity by test-restoring to a temporary database
createdb -h postgres -U pisovereign pisovereign_verify
gunzip -c "$BACKUP_FILE" | pg_restore -h postgres -U pisovereign -d pisovereign_verify 2>&1
INTEGRITY=$(psql -h postgres -U pisovereign -d pisovereign_verify -tAc "SELECT 1;" 2>&1)
dropdb -h postgres -U pisovereign pisovereign_verify

if [ "$INTEGRITY" != "1" ]; then
    echo "ERROR: Database integrity check failed: $INTEGRITY"
    exit 1
fi

echo "Backup verification passed"

Add to cron:

# Verify backup at 3 AM (after 2 AM backup)
0 3 * * * /usr/local/bin/verify-backup.sh || echo "Backup verification failed!" | mail -s "PiSovereign Backup Alert" admin@example.com

Retention Policy

TypeRetentionStorage Estimate
Daily7 days~70 MB
Weekly4 weeks~40 MB
Monthly12 months~120 MB
Total-~230 MB

Cleanup Script

#!/bin/bash
# /usr/local/bin/cleanup-backups.sh

BACKUP_DIR="/backup/pisovereign"

# Remove old daily backups (older than 7 days)
find "$BACKUP_DIR/daily" -name "*.dump.gz" -mtime +7 -delete

# Remove old weekly backups (older than 28 days)
find "$BACKUP_DIR/weekly" -name "*.dump.gz" -mtime +28 -delete

# Remove old monthly backups (older than 365 days)
find "$BACKUP_DIR/monthly" -name "*.dump.gz" -mtime +365 -delete

# Remove old config backups (older than 30 days)
find "$BACKUP_DIR/config" -name "*.toml" -mtime +30 -delete

# Report disk usage
echo "Backup disk usage:"
du -sh "$BACKUP_DIR"/*

Quick Reference

Backup Commands

# Local backup
pisovereign-cli backup \
  --database-url postgres://pisovereign:pisovereign@postgres:5432/pisovereign \
  --output /backup/pisovereign.dump

# S3 backup
pisovereign-cli backup --s3-bucket mybucket --s3-prefix daily/

# Verify backup
pg_restore --list /backup/pisovereign.dump

Restore Commands

# Local restore
pisovereign-cli restore \
  --database-url postgres://pisovereign:pisovereign@postgres:5432/pisovereign \
  --input /backup/pisovereign.dump

# S3 restore
pisovereign-cli restore --s3-bucket mybucket --s3-key daily/pisovereign.dump

Monitoring Backup Health

Add to Prometheus:

# prometheus/rules/backups.yml
groups:
  - name: backups
    rules:
      - alert: BackupMissing
        expr: time() - file_mtime{path="/backup/pisovereign/daily/latest.dump.gz"} > 86400
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Daily backup is missing"
          description: "No backup created in the last 24 hours"

Next Steps