View source for SHADOW METRICS & DISSONANCE RESEARCH SERVICES RUNBOOK

== 🎯 EXECUTIVE SUMMARY ==
Two complementary monitoring services:

* Shadow Metrics (9010): Production-grade, simple monitoring
* Dissonance Research (9011): Experimental, rigorous time-series research platform

----

== 🔧 SERVICE 1: HELIX SHADOW METRICS (PRODUCTION) ==

=== Basic Info ===

* Port: 9010
* Purpose: Basic system monitoring and dissonance tracking
* Status: Production, stable
* Config: <code>/etc/systemd/system/helix-shadow-metrics.service</code>

=== API Endpoints ===
bash
 # Health/status check
 curl <nowiki>http://127.0.0.1:9010/status</nowiki>
 
 # Response:
 {
   "server_time_utc": "2025-11-11T20:01:57.881672Z",
   "version": "1.1.0", 
   "uptime_seconds": 1192,
   "dissonance": {
     "value": 0.12,
     "timestamp": "2025-11-11T19:42:45Z",
     "source": "auto-seed",
     "ip": "127.0.0.1"
   }
 }

=== Troubleshooting ===
bash
 # Check service status
 sudo systemctl status helix-shadow-metrics.service
 
 # Common issue: Permission denied on /opt/helix/
 sudo chown helix:helix /opt/helix/
 
 # Restart service
 sudo systemctl restart helix-shadow-metrics.service
 
 # Check logs
 sudo journalctl -u helix-shadow-metrics.service -n 10

=== Configuration ===

* Seed Value: <code>SEED_VALUE=0.12</code> in env.conf
* Auto-heal: <code>AUTO_HEAL=0</code> (disabled)
* Dissonance Threshold: Δ_t > 0.3 triggers audit trails

----

== 🔬 SERVICE 2: HELIX DISSONANCE RESEARCH (EXPERIMENTAL) ==

=== Basic Info ===

* Port: 9011
* Purpose: Advanced time-series research and experimentation
* Status: Experimental, strict data validation
* Config: <code>/etc/systemd/system/helix-dissonance.service</code>

=== API Endpoints ===
bash
 # Status check
 curl <nowiki>http://127.0.0.1:9011/status</nowiki>
 
 # Submit time data (STRICT SCHEMA REQUIRED)
 curl -X POST <nowiki>http://127.0.0.1:9011/time</nowiki> \
   -H "Content-Type: application/json" \
   -d '{"iso_timestamp": "2025-11-11T20:20:00Z", "source": "test", "timezone": "UTC", "human": "experiment_name"}'
 
 # Retrieve last time entry
 curl <nowiki>http://127.0.0.1:9011/last_time</nowiki>
 
 # API Documentation
 curl <nowiki>http://127.0.0.1:9011/docs</nowiki>

=== Required Data Schema ===
json
 {
   "iso_timestamp": "2025-11-11T20:20:00Z",  // ISO 8601 required
   "timezone": "UTC",                         // Timezone context
   "source": "experiment_source",             // Data provenance
   "human": "Experiment description"          // Human-readable ID
 }

=== Troubleshooting ===
bash
 # Common issue: Port conflict (uses 9011 to avoid conflict with 9010)
 sudo systemctl edit helix-dissonance.service
 # Change port in ExecStart: --port 9011
 
 # Common issue: Truncated ExecStart command
 # Fix in: /etc/systemd/system/helix-dissonance.service.d/z99-execstart.conf
 # Ensure line ends with: --workers 1
 
 # Restart service
 sudo systemctl restart helix-dissonance.service
----

== 🚨 COMMON ISSUES & SOLUTIONS ==

=== Service Won't Start ===
bash
 # Check directory permissions
 sudo ls -la /opt/helix/
 
 # Check if user can access directory  
 sudo -u helix ls -la /opt/helix/
 
 # Fix permissions if needed
 sudo chown helix:helix /opt/helix/

=== Port Conflicts ===
bash
 # Check what's using port 9010/9011
 sudo netstat -tlnp | grep 901
 
 # Change port in service config
 sudo systemctl edit [service-name]

=== Missing Dissonance Data ===
bash
 # Seed file might be missing
 sudo /opt/helix/bin/seed-dissonance.sh
 
 # Check if /run/helix/ exists
 sudo ls -la /run/helix/
----

== 📊 MONITORING & HEALTH CHECKS ==

=== Daily Health Check Script ===
bash
 #!/bin/bash
 # Check both services
 curl -s <nowiki>http://127.0.0.1:9010/status</nowiki> | grep -q '"version":"1.1.0"' && echo "Shadow Metrics: OK" || echo "Shadow Metrics: FAILED"
 curl -s <nowiki>http://127.0.0.1:9011/status</nowiki> | grep -q '"ok":true' && echo "Dissonance Research: OK" || echo "Dissonance Research: FAILED"

=== Key Metrics to Monitor ===

* Service uptime
* Dissonance values
* API response times
* Error rates in logs

----

== 🔄 DEPLOYMENT & MAINTENANCE ==

=== Service Updates ===
bash
 # Reload systemd after config changes
 sudo systemctl daemon-reload
 
 # Restart services
 sudo systemctl restart helix-shadow-metrics.service
 sudo systemctl restart helix-dissonance.service
 
 # Enable on boot
 sudo systemctl enable helix-shadow-metrics.service
 sudo systemctl enable helix-dissonance.service

=== Backup Considerations ===

* Research service stores time series data (check persistence)
* Shadow metrics is stateless
* Backup service configs in <code>/etc/systemd/system/</code>

----

== 🎯 QUICK REFERENCE ==
{| class="wikitable"
!Command
!Purpose
|-
|<code>sudo systemctl status helix-shadow-metrics.service</code>
|Check production service
|-
|<code>sudo systemctl status helix-dissonance.service</code>
|Check research service
|-
|<code>curl <nowiki>http://127.0.0.1:9010/status</nowiki></code>
|Test production API
|-
|<code>curl <nowiki>http://127.0.0.1:9011/status</nowiki></code>
|Test research API
|-
|<code>sudo journalctl -u helix-*-service -n 10</code>
|Check recent logs
|}