Execute. Verify. Rollback. Zero Manual Intervention.

Before executing ANY step, verify preconditions:

1. Resource availability:
   - Disk space sufficient for logs/temp files
   - CPU headroom available (not already at 100%)
   - Memory available for execution

2. Dependency checks:
   - Required binaries present (kubectl, aws, gcloud)
   - Service accounts configured
   - Secrets accessible (SSH keys, API tokens)

3. Conflict detection:
   - No other playbook currently executing (serialized)
   - No maintenance windows active
   - Host not marked as "do not remediate"

If preconditions fail: Abort, report, escalate to human
If preconditions pass: Begin execution

# Each step has timeout
steps:
  - name: clear_temp_files
    timeout: 60s  # Max 60 seconds

  - name: rotate_logs
    timeout: 60s

# If step exceeds timeout: KILL, mark failed, trigger rollback

Execute playbook steps sequentially (not parallel):

Step 1: Clear temp files
  Command: find /tmp -type f -mtime +7 -delete
  Execution: SSH to localhost, run command
  Capture: stdout, stderr, exit code
  Duration: 3.8 seconds
  Result: 100 files deleted, 8.3 GB freed

Step 2: Rotate logs
  Command: logrotate -f /etc/logrotate.conf
  Execution: SSH to localhost, run command
  Capture: stdout, stderr, exit code
  Duration: 1.9 seconds
  Result: 12 log files rotated, 3.1 GB freed

Step 3: Verify space freed
  Command: df -h / | awk 'NR==2 {print $5}' | sed 's/%//'
  Capture: stdout = "72" (72% disk usage)
  Duration: 0.2 seconds
  Result: Disk usage 72% (down from 91%)

After execution completes, verify desired state:

Verification checks:
  1. Metric check: disk_usage < 80%
  2. Service health: HTTP 200 on health endpoint
  3. Error log check: No new errors in /var/log/syslog
  4. Performance check: Disk I/O latency <20ms

Health verification result:
  - Disk usage: 72.1% ✓ (< 80% threshold)
  - HTTP health: 200 OK ✓
  - Error logs: 0 new errors ✓
  - Disk latency: 3ms ✓ (<20ms threshold)

Overall: PASS (all checks passed)

Verification failed: Disk usage 89% (expected <80%)

Rollback triggered:
  1. Identify rollback steps from playbook
  2. Execute rollback steps in reverse order
  3. Verify rollback successful
  4. Report failure to control plane
  5. Escalate to human

Problem: Disk cleanup is one-way (can't restore deleted files)
Action: Escalate to human, incident marked FAILED

Reception:           50ms
Precondition checks: 100ms
Execution:           87s (example: disk cleanup)
Health verification: 20s (3 retries)
Rollback:            0s (verification passed, no rollback needed)

Total: 107 seconds (typical)

{
  "approved_by": "bob.chen@company.com",
  "role": "Remediation Authority",
  "timestamp": "2026-02-10T02:47:52.187Z",
  "ip_address": "203.0.113.42",
  "approval_latency_seconds": 8,
  "incident_id": "inc_2026_02_10_0247",
  "playbook": "postgres_connection_reset",
  "confidence": 0.94
}

{
  "timestamp": "2026-02-10T02:48:00Z",
  "user": "alice.jones@company.com",
  "role": "Observer",
  "action": "approve_playbook",
  "incident_id": "inc_2026_02_10_0247",
  "result": "denied_insufficient_permissions",
  "ip_address": "203.0.113.42"
}

def approve_playbook(user, incident_id):
    # Server-side enforcement (even if client-side bypassed)
    if user.role == "Observer":
        log_failed_authorization(user, incident_id, "approve_playbook")
        raise PermissionError("Insufficient permissions")

    # Permission check passed, proceed
    execute_playbook(incident_id)

# Some steps can rollback, others can't
steps:
  - name: stop_service
    rollback: start_service    # Can rollback

  - name: delete_cache
    rollback: skip             # Can't rollback (files deleted)

  - name: update_config
    rollback: restore_old_config  # Can rollback (config backed up)

# If verification fails:
#   Rollback what's possible (restart service, restore config)
#   Escalate with partial rollback report

steps:
  - name: database_migration
    command: psql -f /migrations/v2.sql
    timeout: 300s  # 5 minutes
    rollback: psql -f /migrations/v2_rollback.sql
    rollback_timeout: 60s  # 1 minute (rollback should be faster)

# Why: Rollback is emergency recovery.
# If it takes too long, system stays broken.