3 Stage Backup Runbook

From Helix Project Wiki

Helix Project – 3‑Stage Backup Runbook

(Designed for the Helix AI Roundtable MediaWiki installation and all supporting services – Qdrant, Ollama, custom Helix daemons, and the surrounding OS configuration.)

📋 Overview

Stage Purpose Scope Frequency Where it runs Where it lands
1️⃣ Daily – “Fast‑Snapshot” Capture the minimum data needed for a quick restore after a minor incident (e.g., config drift, accidental file delete). • Systemd unit files

• Environment files • Recent journal logs • MediaWiki DB dump (MariaDB / SQLite) • Qdrant & Ollama collection metadata (no heavy blobs)

Every night at 02:00 UTC (or local midnight) On the Helix host itself (the same server that runs the services). Local $HOME/backups/daily/ (compressed tar + SHA‑256).
2️⃣ Weekly – “Kitchen‑Sink” Full, immutable copy of everything – suitable for disaster‑recovery (hardware failure, ransomware, cloud‑region loss). All files listed in the Kitchen‑Sink script (full /opt/helix, user home dirs, /etc/systemd, /var/lib/*, logs, model data, snapshots, etc.). Sunday 03:00 UTC Runs on the Helix host; streams a copy to a local off‑site mount (e.g., NFS/SMB) and then to a remote S3‑compatible bucket. $(BACKUP_ROOT)/weekly/ (tarball + SHA‑256) + remote copy.
3️⃣ Test‑Mode – “Configuration/Env Verify” Verify that backup tooling, encryption keys, and restore scripts work before a real outage. No production data is copied; only a synthetic minimal set. • Empty placeholder DB

• Dummy env files • Minimal systemd unit dump

On demand (run manually or via a one‑off cron entry). Runs on the Helix host (or a dedicated “staging” host). $HOME/backups/test/ (tar + SHA‑256).

All three stages produce:

  • A timestamped directory (helix_<stage>_YYYYMMDD_HHMMSS)
  • A compressed tarball (helix_<stage>_YYYYMMDD_HHMMSS.tgz)
  • A SHA‑256 checksum file (SHA256SUMS_YYYYMMDD_HHMMSS.txt)
  • A manifest (INFO.txt) describing what was captured, disk‑space at start/end, and the exact commands executed (audit trail).

1️⃣ DAILY FAST‑SNAPSHOT RUNBOOK

1.1 Prerequisites

Item Recommended Version / Setting
Bash bash ≥5.0
tar GNU tar (supports --zstd if you prefer ZSTD compression).
rsync rsync ≥3.2
mysqldump (or sqlite3) Must be able to run without stopping the MediaWiki DB.
jq For JSON parsing of Qdrant & Ollama health endpoints.
systemd All Helix services are installed as helix‑*.service.
Disk space At least 2 × the expected backup size in $HOME/backups/daily/.
Permissions Run as the helix user or a dedicated backup user with read access to /opt/helix, $HOME, and /etc/systemd.

1.2 Script – ~/bin/helix_daily_backup.sh

#!/usr/bin/env bash
# -------------------------------------------------
# Helix Daily Fast‑Snapshot Backup
# -------------------------------------------------
set -euo pipefail

# ---------- Configuration ----------
STAMP="$(date +%F_%H%M%S)"
BKROOT="${HOME}/backups/daily"
BKDIR="${BKROOT}/helix_daily_${STAMP}"
TAROUT="${BKROOT}/helix_daily_${STAMP}.tgz"
SUMS="${BKROOT}/SHA256SUMS_${STAMP}.txt"

# Services / endpoints
SHIM_URL="http://127.0.0.1:8082"
QDRANT_URL="http://127.0.0.1:6333"
OLLAMA_URL="http://127.0.0.1:11434"

# ---------- Create working dirs ----------
mkdir -p "${BKDIR}/{meta,logs,db_snapshots,env,systemd}"

# ---------- Manifest ----------
{
  echo "=== Helix Daily Backup @ ${STAMP} ==="
  echo "# Disk space (before backup)"
  df -h .
  echo
  echo "# OS / Kernel"
  uname -a
  lsb_release -a 2>/dev/null || true
  echo
  echo "# Running Helix units"
  systemctl list-units 'helix-*.service' --no-legend --plain || true
} > "${BKDIR}/meta/INFO.txt"

# ---------- 1) Service inventory ----------
cp -a /etc/systemd/system/helix-*.service* "${BKDIR}/systemd/" 2>/dev/null || true

# ---------- 2) Health endpoints ----------
curl -fsS "${SHIM_URL}/health"    -o "${BKDIR}/meta/shim_health.json"    || true
curl -fsS "${QDRANT_URL}/collections" -o "${BKDIR}/meta/qdrant_collections.json" || true
curl -fsS "${OLLAMA_URL}/api/tags" -o "${BKDIR}/meta/ollama_tags.json" || true

# ---------- 3) DB dump (fast, no lock) ----------
# Adjust for your DB type – here we assume MariaDB/MySQL.
if command -v mysqldump >/dev/null; then
  mysqldump --single-transaction --quick --skip-lock-tables \
    --user=wikiuser --password="${WIKI_DB_PASS}" wiki > "${BKDIR}/db_snapshots/wiki.sql"
elif command -v sqlite3 >/dev/null; then
  sqlite3 /var/lib/mediawiki/wiki.db .dump > "${BKDIR}/db_snapshots/wiki.sql"
fi

# ---------- 4) Recent journal logs ----------
{
  echo "## helix-upsert-shim (last 200 lines)"
  journalctl -u helix-upsert-shim -n 200 --no-pager || true
  echo
  echo "## helix-image-intake (last 200 lines)"
  journalctl -u helix-image-intake -n 200 --no-pager || true
  echo
  echo "## helix-runner (last 200 lines)"
  journalctl -u helix-runner -n 200 --no-pager || true
} > "${BKDIR}/logs/journal_helix_services.txt"

# ---------- 5) Package ----------
tar -C "${BKDIR}/.." -czf "${TAROUT}" "$(basename "${BKDIR}")"

# ---------- 6) Checksums ----------
sha256sum "${TAROUT}" | tee "${SUMS}" >/dev/null

# ---------- Completion info ----------
{
  echo "=== DONE ==="
  echo "Backup size: $(du -h "${TAROUT}" | cut -f1)"
  echo "SHA256: $(cat "${SUMS}")"
} | tee -a "${BKDIR}/meta/INFO.txt"

exit 0

Make the script executable: 

chmod +x ~/bin/helix_daily_backup.sh

1.3 Cron entry (host‑side)

<# Helix daily fast‑snapshot – runs as the dedicated backup user
0 2 * * * /bin/helix_daily_backup.sh >> $HOME/backups/daily/cron.log 2>&1

All output is also written to the backup manifest, satisfying audit‑ability.

1.4 Verification after run

  1. Checksum – sha256sum -c $SUMS should report OK.
  2. Manifest – open ${BKDIR}/meta/INFO.txt and confirm that every section contains data (no “failed” messages).
  3. Size sanity – du -h ${BKDIR} should be < 1 GB for most installations; adjust retention if it grows.

1️⃣ Restore (daily) – Quick steps

# 1) Verify checksum of the tarball
sha256sum -c ${HOME}/backups/daily/SHA256SUMS_YYYYMMDD_HHMMSS.txt

# 2) Extract to a temporary location
mkdir -p /tmp/helix_restore && tar -C /tmp/helix_restore -xzf \
    ${HOME}/backups/daily/helix_daily_YYYYMMDD_HHMMSS.tgz

# 3) Restore systemd files (if needed)
sudo cp -a /tmp/helix_restore/systemd/helix-*.service* /etc/systemd/system/
sudo systemctl daemon-reload

# 4) Reload DB (if you need to roll back)
#   Example for MySQL:
mysql -u wikiuser -p"${WIKI_DB_PASS}" wiki < /tmp/helix_restore/db_snapshots/wiki.sql

# 5) Reload MediaWiki config
sudo systemctl restart helix-upsert-shim.service helix-runner.service

NOTE: Because the daily backup never stops services, it is only safe for metadata and configuration rollback, not for full data loss.

2️⃣ WEEKLY KITCHEN‑SINK BACKUP RUNBOOK

2.1 Prerequisites

Requirement Details
Bash (Linux) or PowerShell (Windows) Both scripts are provided; pick the one that matches the host OS.
Remote off‑site mount e.g., /mnt/offsite_helix (NFS, SMB, or an encrypted LUKS volume).
S3‑compatible bucket awscli or rclone configured with server‑side encryption (SSE‑S3 or SSE‑KMS).
SSH key‑pair Password‑less access to any remote host used for off‑site copy.
rsync, scp, ssh, tar, sha256sum All must be present.
Disk quota Keep at least 3 × the expected weekly size in ${BACKUP_ROOT}.
Encryption (optional) If you want encrypted archives, replace tar -czf with tar -I 'zstd -9 -e' or pipe through gpg --symmetric.

2.2 Script – Linux (bash)

Save as ~/bin/helix_weekly_kitchen_sink.sh 

#!/usr/bin/env bash
# -------------------------------------------------
# Helix Weekly “Kitchen‑Sink” Full Backup
# -------------------------------------------------
set -euo pipefail

# ---------- Global config ----------
STAMP="$(date +%F_%H%M%S)"
BKROOT="${HOME}/backups/weekly"
BKDIR="${BKROOT}/helix_kitchen_${STAMP}"
TAROUT="${BKROOT}/helix_kitchen_${STAMP}.tgz"
SUMS="${BKROOT}/SHA256SUMS_${STAMP}.txt"
OFFSITE_MOUNT="/mnt/offsite_helix"          # <‑‑ change to your mount point
REMOTE_BUCKET="s3://helix-backups/weekly"   # <‑‑ bucket configured in ~/.aws/credentials

# Service endpoints (same as daily)
SHIM_URL="http://127.0.0.1:8082"
QDRANT_URL="http://127.0.0.1:6333"
OLLAMA_URL="http://127.0.0.1:11434"

# ---------- Working dirs ----------
mkdir -p "${BKDIR}"/{meta,logs,etc_systemd,opt_helix,home_snap,db_snapshots,ollama,qdrant}

# ---------- Manifest ----------
{
  echo "=== Helix Kitchen‑Sink Backup @ ${STAMP} ==="
  echo "# Disk space (pre‑backup)"
  df -h .
  echo
  echo "# OS / Kernel"
  uname -a
  lsb_release -a 2>/dev/null || true
  echo
  echo "# All Helix systemd units"
  systemctl list-units 'helix-*.service' --all --no-legend --plain || true
} > "${BKDIR}/meta/INFO.txt"

# ---------- 0) Quick space check ----------
df -h . >> "${BKDIR}/meta/INFO.txt"

# ---------- 1) System inventory ----------
cp -a /etc/systemd/system/helix-*.service* "${BKDIR}/etc_systemd/" 2>/dev/null || true

# ---------- 2) Health endpoints ----------
curl -fsS "${SHIM_URL}/health"    -o "${BKDIR}/meta/shim_health.json"    || true
curl -fsS "${QDRANT_URL}/collections" -o "${BKDIR}/meta/qdrant_collections.json" || true
curl -fsS "${OLLAMA_URL}/api/tags" -o "${BKDIR}/ollama/models_tags.json" || true

# ---------- 3) Qdrant snapshots (all collections) ----------
if curl -fsS "${QDRANT_URL}/collections" -o "${BKDIR}/meta/qdrant_collections.json"; then
  if command -v jq >/dev/null; then
    mapfile -t COLS < <(jq -r '.result.collections[].name' "${BKDIR}/meta/qdrant_collections.json")
  else
    mapfile -t COLS < <(grep -oE '"name":"[^"]+"' "${BKDIR}/meta/qdrant_collections.json" | \
                      sed 's/"name":"//;s/"$//')
  fi
  for c in "${COLS[@]}"; do
    curl -fsS -X POST "${QDRANT_URL}/collections/${c}/snapshots" \
         -o "${BKDIR}/db_snapshots/${c}_snapshot_${STAMP}.json" || true
  done
fi

# ---------- 4) Application tree ----------
[ -d /opt/helix ] && rsync -a /opt/helix/ "${BKDIR}/opt_helix/" || true

# ---------- 5) Home‑side bits ----------
for d in \
  "$HOME/apps" \
  "$HOME/trust" \
  "$HOME/save" \
  "$HOME/helix-logs" \
  "$HOME/shim-venv" \
  "$HOME/helix_endpoints.env" \
  "$HOME/helix_envsnap_"* \
  "$HOME/helix_upsert_shim.py" \
  "$HOME/helix_env_primer.sh"; do
  [ -e "$d" ] && rsync -a "$d" "${BKDIR}/home_snap/" || true
done

# ---------- 6) Recent journal logs ----------
{
  echo "## helix-upsert-shim (last 200)"
  journalctl -u helix-upsert-shim -n 200 --no-pager || true
  echo
  echo "## helix-image-intake (last 200)"
  journalctl -u helix-image-intake -n 200 --no-pager || true
  echo
  echo "## helix-runner (last 200)"
  journalctl -u helix-runner -n 200 --no-pager || true
} > "${BKDIR}/logs/journal_helix_services.txt"

# ---------- 7) Archive -------------------------------------------------
tar -C "${BKDIR}/.." -czf "${TAROUT}" "$(basename "${BKDIR}")"

# ---------- 8) Checksums & final size ----------
sha256sum "${TAROUT}" | tee "${SUMS}" >/dev/null
du -h "${TAROUT}" | tee -a "${BKDIR}/meta/INFO.txt"

# ---------- 9) Off‑site copy (local mount + S3) ----------
# a) Local off‑site mount (if mounted)
if mountpoint -q "${OFFSITE_MOUNT}"; then
  cp "${TAROUT}" "${OFFSITE_MOUNT}/"
  cp "${SUMS}"   "${OFFSITE_MOUNT}/"
fi

# b) Remote S3 bucket (requires awscli or rclone)
if command -v aws >/dev/null; then
  aws s3 cp "${TAROUT}" "${REMOTE_BUCKET}/" --storage-class STANDARD_IA
  aws s3 cp "${SUMS}"   "${REMOTE_BUCKET}/"
elif command -v rclone >/dev/null; then
  rclone copy "${TAROUT}" "myremote:helix/weekly"
  rclone copy "${SUMS}"  "myremote:helix/weekly"
fi

echo "=== DAILY BACKUP COMPLETE ==="

Make the script executable: 

chmod +x ~/bin/helix_daily_backup.sh

2.3 Cron entry (Linux host)

# Daily fast‑snapshot – runs as the backup user
0 2 * * * /home/helix/bin/helix_daily_backup.sh >> $HOME/backups/daily/cron.log 2>&1

The >> appends log output to a single daily log file; the manifest inside the tarball provides a full audit record.

2️⃣ WEEKLY KITCHEN‑SINK BACKUP RUNBOOK

2.1 Prerequisites

Requirement Detail
PowerShell 7+ (for Windows hosts) or Bash (Linux) – the same script works on both with minor syntax changes.
SSH key with password‑less login to the remote host (if you stream the backup to another machine).
Rclone / AWS‑CLI configured for your off‑site bucket (~/.aws/credentials or rclone.conf).
GPG (optional) – for encrypting the tarball before it leaves the host.
rsync – for fast local copying of large model directories.
Sufficient network bandwidth – a full Helix install (including Ollama models) can be > 50 GB.
Retention policy – keep at least 4 weekly archives locally; older ones are pruned automatically.

2.2 Bash version (Linux)

Save as ~/bin/helix_weekly_kitchen_sink.sh 

#!/usr/bin/env bash
# -------------------------------------------------
# Helix Weekly Kitchen‑Sink Backup (Full Disaster‑Recovery)
# -------------------------------------------------
set -euo pipefail

# ---------- Config ----------
STAMP="$(date +%F_%H%M%S)"
BKROOT="${HOME}/backups/weekly"
BKDIR="${BKROOT}/helix_kitchen_${STAMP}"
TAROUT="${BKROOT}/helix_kitchen_${STAMP}.tgz"
SUMS="${BKROOT}/SHA256SUMS_${STAMP}.txt"

# Remote off‑site mount (NFS/SMB) – edit to match your environment
OFFSITE_MOUNT="/mnt/offsite_helix"
REMOTE_BUCKET="s3://helix-backups/weekly"   # S3‑compatible bucket

# ---------- Create dirs ----------
mkdir -p "${BKDIR}/{meta,logs,systemd,opt_helix,db_snapshots,home_snap}"

# ---------- Manifest ----------
{
  echo "=== Helix Kitchen‑Sink Backup @ ${STAMP} ==="
  df -h .
  uname -a
  lsb_release -a 2>/dev/null || true
  echo
  echo "# All Helix systemd units"
  systemctl list-units 'helix-*.service' --all --no-legend --plain || true
} > "${BKDIR}/meta/INFO.txt"

# ---------- 1) Systemd unit files ----------
cp -a /etc/systemd/system/helix-*.service* "${BKDIR}/systemd/" 2>/dev/null || true

# ---------- 2) Full app tree ----------
if [ -d /opt/helix ]; then
  rsync -a /opt/helix/ "${BKDIR}/opt_helix/" || true
fi

# ---------- 3) User home data ----------
for d in \
  "$HOME/apps" \
  "$HOME/trust" \
  "$HOME/save" \
  "$HOME/helix-logs" \
  "$HOME/shim-venv" \
  "$HOME/helix_endpoints.env" \
  "$HOME/helix_envsnap_"*; do
  [ -e "$d" ] && rsync -a "$d" "${BKDIR}/home_snap/" || true
done

# ---------- 4) Qdrant & Ollama snapshots (binary) ----------
# (Same logic as daily, but we also copy the snapshot files themselves.)
QDRANT_SNAP_ROOT="${HOME}/.qdrant/snapshots"
Ollama_MODEL_ROOT="${HOME}/.ollama"

if [ -d "${QDRANT_SNAP_ROOT}" ]; then
  rsync -a "${QDRANT_SNAP_ROOT}/" "${BKDIR}/db_snapshots/qdrant/" || true
fi
if [ -d "${Ollama_MODEL_ROOT}" ]; then
  rsync -a "${Ollama_MODEL_ROOT}/" "${BKDIR}/ollama_models/" || true
fi

# ---------- 5) Recent logs ----------
journalctl -u helix-*.service -n 500 --no-pager > "${BKDIR}/logs/journal_recent.txt" || true

# ---------- 6) Create archive (gzip, ZSTD, or GPG) ----------
# Choose one of the following compression methods:
#   tar -czf    – gzip (default)
#   tar --zstd – ZSTD (faster, better compression)
#   tar -c | gpg --symmetric --cipher-algo AES256 -o <outfile>
tar -C "${BKDIR}/.." -czf "${TAROUT}" "$(basename "${BKDIR}")"

# ---------- 7) Checksums ----------
sha256sum "${TAROUT}" | tee "${SUMS}" >/dev/null

# ---------- 8) Off‑site copy ----------
# a) Local mount copy (if available)
if mountpoint -q "${OFFSITE_MOUNT}"; then
  cp "${TAROUT}" "${OFFSITE_MOUNT}/"
  cp "${SUMS}"   "${OFFSITE_MOUNT}/"
fi

# b) Remote S3 / Rclone copy
if command -v aws >/dev/null; then
  aws s3 cp "${TAROUT}" "${REMOTE_BUCKET}/" --storage-class STANDARD_IA
  aws s3 cp "${SUMS}"   "${REMOTE_BUCKET}/"
elif command -v rclone >/dev/null; then
  rclone copy "${TAROUT}" "myremote:helix/weekly"
  rclone copy "${SUMS}"  "myremote:helix/weekly"
fi

# ---------- 9) Prune old weekly archives (keep last 4) ----------
find "${BKROOT}" -maxdepth 1 -name "helix_kitchen_*.tgz" -mtime +28 -delete || true
find "${BKROOT}" -maxdepth 1 -name "SHA256SUMS_*.txt" -mtime +28 -delete || true

echo "=== WEEKLY KITCHEN‑SINK BACKUP COMPLETE ==="

Make executable 

chmod +x ~/bin/helix_weekly_backup.sh

2.3 PowerShell version (Windows‑based Helix node)

If you ever need to back up a Windows‑host that runs a copy of Helix (e.g., for a hybrid on‑prem / Azure VM), the script you posted is already a solid baseline. Below is a trimmed‑down version that follows the same three‑stage philosophy and can be called from a Windows Task Scheduler or a Linux‑style cron via WSL. 

# File: C:\Helix\Scripts\Helix_Weekly_KitchenSink.ps1
param(
    [string]$RemoteHost   = "148.113.222.171",
    [string]$RemoteUser   = "aiadmin",
    [string]$BackupRoot   = "C:\HelixBackups\Weekly",
    [switch]$IncludeModels
)

function Log { param([string]$msg,[string]$c='White') Write-Host $msg -ForegroundColor $c }

$ts       = Get-Date -Format "yyyyMMdd_HHmmss"
$destDir  = Join-Path $BackupRoot "helix_complete_$ts"
New-Item -ItemType Directory -Path $destDir -Force | Out-Null
Log "🗄️  Destination: $destDir" "Green"

# --- Verify SSH connectivity -------------------------------------------------
try {
    ssh -o BatchMode=yes -o ConnectTimeout=10 "$RemoteUser@$RemoteHost" "echo ok" | Out-Null
    Log "✅ SSH connection OK" "Green"
} catch {
    Log "❌ Unable to reach $RemoteHost" "Red"
    exit 1
}

# --- Define what to copy ------------------------------------------------------
$targets = @(
    "/opt/helix"
    "/etc/systemd/system/helix-*.service"
    "/home/$RemoteUser/apps"
    "/home/$RemoteUser/trust"
    "/home/$RemoteUser/save"
    "/home/$RemoteUser/helix-logs"
)

if ($IncludeModels) {
    $targets += "/home/$RemoteUser/.ollama"
}

# --- Pull each target via SCP -------------------------------------------------
foreach ($t in $targets) {
    $src = "$RemoteUser@$RemoteHost:`"$t`""
    Log "📥 Pulling $t ..." "Cyan"
    scp -r $src $destDir
}

# --- Create a single archive --------------------------------------------------
$archive = Join-Path $BackupRoot "helix_weekly_$ts.zip"
Compress-Archive -Path "$destDir\*" -DestinationPath $archive -Force
Log "🗜️  Archive created: $archive" "Cyan"

# --- Optional GPG encryption -------------------------------------------------
# If you want encryption:
# gpg --symmetric --cipher-algo AES256 -o "$archive.gpg" $archive

# --- Upload to Azure Blob / S3 ------------------------------------------------
# Example using Azure AzCopy:
# azcopy copy $archive "https://myaccount.blob.core.windows.net/helix/weekly?$SAS" --blob-type BlockBlob

Log "✅ Weekly kitchen‑sink backup finished" "Green"

Scheduling (Windows Task Scheduler)

  • Action: powershell.exe -ExecutionPolicy Bypass -File "C:\Helix\Scripts\Helix_Weekly_KitchenSink.ps1" -IncludeModels
  • Trigger: Weekly on Sunday at 02:00.

2.4 Cron entry (Linux host)

# Weekly full kitchen‑sink – runs as the backup user
0 3 * * 0 /home/helix/bin/helix_weekly_kitchen_sink.sh >> $HOME/backups/weekly/cron.log 2>&1

Why Sunday at 03:00? Gives a window after daily backups have completed, and avoids peak business hours.

3️⃣ TESTING / VALIDATION PROCEDURES

  1. Initial dry‑run – Set set -x (bash) or $DebugPreference = 'Continue' (PowerShell) to see every command.
  2. Checksum validation – After each backup, run sha256sum -c on the generated checksum file.
  3. Restore rehearsal – Periodically (e.g., monthly) pick the newest weekly archive, extract it on a staging machine, and perform a full restore test (including starting Helix services).
  4. Log aggregation – Centralize the cron.log and PowerShell task logs in a log‑management system (e.g., Loki, ELK) for long‑term monitoring.

4️⃣ RETENTION & HOUSEKEEPING

  • Daily: Keep 7 days (find $HOME/backups/daily -type f -mtime +7 -delete).
  • Weekly: Keep 4 weeks (find $HOME/backups/weekly -type f -mtime +28 -delete).
  • Monthly (optional): Move the oldest weekly archive to a cold storage bucket (Glacier, Azure Archive).

These policies can be encoded in a single cleanup script or added to each backup script as shown.

5️⃣ SECURITY & COMPLIANCE NOTES

Concern Mitigation
Data at rest Store backups on encrypted disks (LUKS, BitLocker) and encrypt archives with GPG before off‑site upload.
Data in transit Use SSH, TLS (HTTPS health endpoints), and S3‑HTTPS.
Access control Restrict the backup user to only the backup directories (chmod 700).
Key management Rotate SSH keys and GPG passphrases every 90 days.
Audit Every backup tarball contains a meta/INFO.txt manifest with timestamps, system info, and any error messages – this satisfies the Transparency principle of the Trustworthy AI charter.

QUICK REFERENCE TABLE

Stage Script (Linux) Script (PowerShell) Cron / Scheduler Key Points
Daily helix_daily_backup.sh – (not needed) 0 2 * * * Fast, no service stop, config + logs + DB metadata
Weekly (Full) helix_weekly_kitchen_sink.sh Helix_Weekly_KitchenSink.ps1 0 3 * * 0 (Linux) / Weekly Task (Win) Full app tree, user data, model dirs, off‑site copy, retention pruning
Test restore_daily.sh (example) Restore-Weekly.ps1 – Verify checksum, extract, copy systemd files, reload services

Final Checklist before production deployment

  • Scripts placed in version‑controlled repository (e.g., Git) and reviewed.
  • Execution permissions set (chmod +x).
  • Cron / Task Scheduler entries created for the correct user.
  • Off‑site mount and S3 bucket reachable; test with a small dummy file.
  • GPG keys imported and passphrase stored in a password manager (not in script).
  • Retention policies implemented (find … -delete).
  • Documentation of the backup locations, encryption keys, and bucket names stored in a separate, secured vault.

With these scripts and policies in place, your Helix deployment will have daily rapid‑recovery capability for configuration drift and weekly a full disaster‑recovery snapshot that can survive a total site failure. All steps are transparent, auditable, and align with the trustworthy AI ethos you’re championing. Happy backing‑up!

Retrieved from "https://helixprojectai.com:443/wiki/index.php?title=3_Stage_Backup_Runbook&oldid=249"