diff --git a/HIGH-AVAILABILITY.md b/HIGH-AVAILABILITY.md new file mode 100644 index 000000000..10a2b4762 --- /dev/null +++ b/HIGH-AVAILABILITY.md @@ -0,0 +1,487 @@ +# 🏗️ Moltbot 高可用性和自动化指南 + +**版本**: v2.2 +**最后更新**: 2026-01-29 + +--- + +## 📋 高可用性 (HA) 架构 + +### 架构概览 + +``` + ┌───────────────────┐ + │ Virtual IP │ + │ (38.14.254.100) │ + └────────┬───────────┘ + │ + ┌────────────┴────────────┐ + │ │ + ┌──────▼──────┐ ┌──────▼──────┐ + │ Master │ │ Backup │ + │ Server │ │ Server │ + │ │ │ │ + │ Gateway │ │ Gateway │ + │ PostgreSQL │ │ PostgreSQL │ + │ Monitoring │ │ Monitoring │ + └─────────────┘ └─────────────┘ + │ │ + └────────────┬────────────┘ + │ + ┌────────────▼───────────┐ + │ Shared Storage │ + │ (Optional) │ + └────────────────────────┘ +``` + +--- + +## 🚀 快速开始 + +### 一键部署新服务器 + +在全新的服务器上运行: + +```bash +# 方法 1: 使用 curl +curl -fsSL https://raw.githubusercontent.com/flowerjunjie/moltbot/main/deploy-oneclick.sh | bash + +# 方法 2: 使用 git +git clone https://github.com/flowerjunjie/moltbot.git /opt/moltbot +cd /opt/moltbot +bash deploy-oneclick.sh +``` + +### 远程部署服务器 + +从本地机器部署到远程服务器: + +```bash +# Linux/Mac +bash auto-deploy-server.sh root@192.168.1.100 + +# Windows +auto-deploy-server.bat root@192.168.1.100 +``` + +--- + +## 🔧 高可用性组件 + +### 1. Keepalived (虚拟 IP) + +**功能**: 自动故障转移和虚拟 IP 管理 + +**安装**: +```bash +apt-get install keepalived +``` + +**配置文件**: `/etc/keepalived/keepalived.conf` +```conf +vrrp_script chk_moltbot_gateway { + script "curl -f http://localhost:18789 || exit 1" + interval 2 + weight 2 +} + +vrrp_instance VI_MOLTBOT { + state MASTER + interface eth0 + virtual_router_id 51 + priority 100 + advert_int 1 + + authentication { + auth_type PASS + auth_pass moltbot2024 + } + + virtual_ipaddress { + 38.14.254.100/24 + } + + track_script { + chk_moltbot_gateway + } +} +``` + +**状态检查**: +```bash +systemctl status keepalived +ip addr show eth0 | grep 38.14.254.100 +``` + +### 2. 自动故障转移 + +**脚本**: `/usr/local/bin/moltbot-failover.sh` + +**功能**: +- 健康检查(每 10 秒) +- 自动重启失败的服务 +- 故障计数和阈值 +- 日志记录 + +**服务**: `moltbot-failover.service` + +**启用**: +```bash +systemctl enable moltbot-failover +systemctl start moltbot-failover +``` + +**查看日志**: +```bash +journalctl -u moltbot-failover -f +cat /var/log/moltbot-failover.log +``` + +### 3. PostgreSQL 流复制 + +**配置**: `/etc/postgresql/14/main/conf.d/replication.conf` + +**设置主服务器**: +```sql +-- 创建复制用户 +CREATE USER replicator WITH REPLICATION ENCRYPTED PASSWORD 'replicator_pass'; + +-- 配置复制槽 +SELECT * FROM pg_create_physical_replication_slot('replica_slot'); +``` + +**设置从服务器**: +```bash +# 在从服务器上 +pg_basebackup -h master-server -D /var/lib/postgresql/data -P -U replicator --wal-method=stream + +# 配置 recovery.conf +standby_mode = on +primary_conninfo = 'host=master-server port=5432 user=replicator' +restore_command = 'cp /var/lib/postgresql/archive/%f %p' +``` + +### 4. 灾难恢复备份 + +**脚本**: `/usr/local/bin/moltbot-dr-backup.sh` + +**备份内容**: +- PostgreSQL 完整转储 +- 配置文件 +- Docker 卷数据 +- 系统包列表 +- 防火墙规则 + +**运行备份**: +```bash +/usr/local/bin/moltbot-dr-backup.sh +``` + +**备份位置**: `/opt/moltbot-backup/disaster-recovery/` + +**自动备份**: 每周日凌晨 3 点 + +--- + +## 🤖 自动化工具 + +### 1. 自动部署工具 + +**文件**: `auto-deploy-server.sh` (Linux) / `auto-deploy-server.bat` (Windows) + +**功能**: +- 自动安装所有依赖 +- 配置数据库 +- 部署监控栈 +- 设置防火墙 +- 配置自动化任务 + +**使用**: +```bash +# 部署到新服务器 +bash auto-deploy-server.sh root@192.168.1.100 +``` + +### 2. 一键部署脚本 + +**文件**: `deploy-oneclick.sh` + +**场景**: 在全新的服务器上运行 + +**使用**: +```bash +# SSH 到服务器 +ssh root@your-server + +# 运行部署 +curl -fsSL https://raw.githubusercontent.com/flowerjunjie/moltbot/main/deploy-oneclick.sh | bash +``` + +**部署时间**: 约 5-10 分钟 + +### 3. 容器编排支持 + +**文件**: `docker-compose-full.yml` + +**包含服务**: +- Moltbot Gateway +- Database API +- PostgreSQL +- Redis +- Prometheus +- Grafana +- Node Exporter +- Metrics Exporter +- Log Analyzer +- Nginx + +**启动**: +```bash +docker-compose -f docker-compose-full.yml up -d +``` + +--- + +## 📊 监控和告警 + +### 服务端口 + +| 服务 | 端口 | 说明 | +|------|------|------| +| Database API | 18800 | REST API | +| Metrics | 9101 | Prometheus 指标 | +| Log Analyzer | 9102 | 日志分析 API | +| Prometheus | 9090 | 指标采集 | +| Grafana | 3000 | 可视化 | + +### 健康检查端点 + +```bash +# Database API +curl http://localhost:18800/api/health + +# Metrics +curl http://localhost:9101/metrics + +# Log summary +curl http://localhost:9102/api/logs/summary + +# Service status +curl http://localhost:18800/api/devices +``` + +--- + +## 🛠️ 维护操作 + +### 日常维护 + +**检查服务状态**: +```bash +# 所有 Moltbot 服务 +systemctl status moltbot-* + +# Docker 容器 +docker ps + +# 监控栈 +cd /opt/moltbot-monitoring && docker-compose ps +``` + +**查看日志**: +```bash +# 服务日志 +journalctl -u moltbot-db-api -f +journalctl -u moltbot-failover -f + +# 应用日志 +tail -f /var/log/moltbot-failover.log +``` + +### 备份操作 + +**手动备份**: +```bash +# 数据库备份 +/usr/local/bin/moltbot-backup-auto.sh + +# 灾难恢复备份 +/usr/local/bin/moltbot-dr-backup.sh +``` + +**恢复数据库**: +```bash +# 列出备份 +ls -lh /opt/moltbot-backup/database/daily/ + +# 恢复最新备份 +gunzip -c /opt/moltbot-backup/database/daily/moltbot_latest.sql.gz | psql -d moltbot +``` + +### 故障排除 + +**服务无法启动**: +```bash +# 检查端口占用 +netstat -tlnp | grep + +# 检查日志 +journalctl -u -n 50 + +# 重启服务 +systemctl restart +``` + +**Keepalived 问题**: +```bash +# 检查配置 +keepalived -t + +# 查看日志 +journalctl -u keepalived -f + +# 检查虚拟 IP +ip addr show eth0 +``` + +--- + +## 🔐 安全配置 + +### 防火墙规则 + +**查看当前规则**: +```bash +iptables -L -n -v +``` + +**添加规则**: +```bash +iptables -A INPUT -p tcp --dport 18789 -s 192.168.1.0/24 -j ACCEPT +netfilter-persistent save +``` + +### 安全建议 + +1. **使用密钥认证**: 禁用密码登录 +2. **配置 fail2ban**: 防止暴力攻击 +3. **定期更新**: `apt-get update && apt-get upgrade` +4. **监控日志**: 定期检查异常访问 + +--- + +## 📈 性能优化 + +### 系统优化 + +**运行优化脚本**: +```bash +/usr/local/bin/moltbot-optimize.sh +``` + +**优化项目**: +- 网络参数调优 +- PostgreSQL 配置优化 +- Docker 资源限制 +- 日志轮转配置 + +### 性能监控 + +**查看系统指标**: +```bash +# CPU +top -bn1 | grep "Cpu(s)" + +# 内存 +free -h + +# 磁盘 +df -h + +# 负载 +cat /proc/loadavg +``` + +--- + +## 🚨 应急响应 + +### 服务全部宕机 + +1. **检查服务器状态** + ```bash + ping + ssh root@ "systemctl status moltbot-*" + ``` + +2. **启动关键服务** + ```bash + systemctl start moltbot-db-api + systemctl start moltbot-gateway + ``` + +3. **切换到备用服务器**(如果配置了 HA) + ```bash + # 备用服务器会自动提升为主服务器 + # 虚拟 IP 会自动迁移 + ``` + +### 数据库损坏 + +1. **从备份恢复** + ```bash + gunzip -c /opt/moltbot-backup/disaster-recovery/pg_all_*.sql.gz | psql + ``` + +2. **检查数据完整性** + ```bash + psql -d moltbot -c "SELECT COUNT(*) FROM conversations;" + psql -d moltbot -c "SELECT COUNT(*) FROM devices;" + ``` + +### 网络问题 + +1. **检查网络连接** + ```bash + ping 8.8.8.8 + traceroute 8.8.8.8 + ``` + +2. **检查防火墙** + ```bash + iptables -L -n + ufw status + ``` + +--- + +## 📚 相关文档 + +- `DEPLOYMENT-COMPLETE.md` - 完整部署指南 +- `EXTENSIONS.md` - 扩展功能文档 +- `ROADMAP.md` - 功能路线图 +- `docker-compose-full.yml` - 容器编排配置 + +--- + +## 🎯 最佳实践 + +1. **定期测试备份恢复** + - 每月测试一次灾难恢复流程 + - 验证备份完整性 + +2. **监控告警** + - 配置邮件或 Webhook 告警 + - 设置合理的告警阈值 + +3. **文档更新** + - 记录所有配置更改 + - 维护操作手册 + +4. **容量规划** + - 监控资源使用趋势 + - 提前规划扩容 + +--- + +**🎉 高可用性和自动化配置完成!** diff --git a/auto-deploy-server.bat b/auto-deploy-server.bat new file mode 100644 index 000000000..c3120325c --- /dev/null +++ b/auto-deploy-server.bat @@ -0,0 +1,63 @@ +@echo off +chcp 65001 >nul +title Moltbot Automated Server Deployment + +setlocal enabledelayedexpansion + +echo ======================================== +echo Moltbot Server Auto Deployment +echo ======================================== +echo. + +if "%~1"=="" ( + echo Usage: auto-deploy-server.bat [server-address] + echo. + echo Examples: + echo auto-deploy-server.bat root@192.168.1.100 + echo auto-deploy-server.bat user@example.com + echo. + pause + exit /b 1 +) + +set "SERVER=%~1" +set "SCRIPT=%~dp0auto-deploy-server.sh" + +echo Target server: %SERVER% +echo. + +if not exist "%SCRIPT%" ( + echo ERROR: auto-deploy-server.sh not found + echo This script requires the bash deployment script. + pause + exit /b 1 +) + +echo Checking for WSL or Git Bash... +where wsl.exe >nul 2>&1 +if %errorlevel%==0 ( + echo Using WSL to run deployment script... + wsl.exe bash "%SCRIPT%" "%SERVER%" + goto end +) + +where bash.exe >nul 2>&1 +if %errorlevel%==0 ( + echo Using Git Bash to run deployment script... + bash.exe "%SCRIPT%" "%SERVER%" + goto end +) + +echo ERROR: No bash interpreter found +echo Please install WSL or Git for Windows +echo. +pause +exit /b 1 + +:end +echo. +echo ======================================== +echo Deployment Complete! +echo ======================================== +echo. +pause diff --git a/auto-deploy-server.sh b/auto-deploy-server.sh new file mode 100644 index 000000000..04d12a68b --- /dev/null +++ b/auto-deploy-server.sh @@ -0,0 +1,218 @@ +#!/bin/bash +# +# Moltbot Automated Server Deployment +# Deploys complete Moltbot stack to a new server +# + +set -e + +# Color output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' # No Color + +print_status() { + echo -e "${GREEN}[INFO]${NC} $1" +} + +print_error() { + echo -e "${RED}[ERROR]${NC} $1" +} + +print_warning() { + echo -e "${YELLOW}[WARNING]${NC} $1" +} + +# Check if target server is provided +if [ -z "$1" ]; then + print_error "Usage: $0 [ssh-port]" + echo "" + echo "Example:" + echo " $0 root@192.168.1.100" + echo " $0 user@example.com 2222" + exit 1 +fi + +SERVER=$1 +SSH_PORT=${2:-22} + +print_status "Starting Moltbot deployment to $SERVER..." +echo "" + +# Test SSH connection +print_status "Testing SSH connection..." +if ! ssh -p $SSH_PORT -o ConnectTimeout=10 $SERVER "echo 'Connection successful'"; then + print_error "Cannot connect to $SERVER" + exit 1 +fi + +# Step 1: Update system +print_status "[1/10] Updating system packages..." +ssh -p $SSH_PORT $SERVER "apt-get update -qq && apt-get upgrade -y -qq" + +# Step 2: Install dependencies +print_status "[2/10] Installing dependencies..." +ssh -p $SSH_PORT $SERVER "apt-get install -y -qq curl git wget python3 python3-pip postgresql postgresql-contrib nginx docker.io docker-compose nodejs npm build-essential" + +# Step 3: Clone repository +print_status "[3/10] Cloning Moltbot repository..." +ssh -p $SSH_PORT $SERVER "cd /opt && rm -rf moltbot && git clone https://github.com/flowerjunjie/moltbot.git moltbot" + +# Step 4: Install Python dependencies +print_status "[4/10] Installing Python packages..." +ssh -p $SSH_PORT $SERVER "pip3 install -q psycopg2-binary psutil" + +# Step 5: Setup database +print_status "[5/10] Setting up PostgreSQL database..." +ssh -p $SSH_PORT $SERVER "sudo -u postgres psql -c 'CREATE DATABASE moltbot;' && sudo -u postgres psql -c \"CREATE USER root WITH SUPERUSER;\" && sudo -u postgres psql -c 'ALTER USER root WITH PASSWORD;'\"" + +# Step 6: Create database tables +print_status "[6/10] Creating database tables..." +ssh -p $SSH_PORT $SERVER "psql -d moltbot << 'SQL' +-- Conversations table +CREATE TABLE IF NOT EXISTS conversations ( + id SERIAL PRIMARY KEY, + device_id VARCHAR(100) NOT NULL, + session_id VARCHAR(100) NOT NULL, + role VARCHAR(20) NOT NULL, + content TEXT NOT NULL, + model VARCHAR(100), + tokens INTEGER, + created_at TIMESTAMP DEFAULT NOW() +); + +-- Devices table +CREATE TABLE IF NOT EXISTS devices ( + device_name VARCHAR(100) UNIQUE NOT NULL, + device_type VARCHAR(50), + ip_address VARCHAR(50), + last_seen TIMESTAMP DEFAULT NOW(), + status VARCHAR(20) DEFAULT 'online' +); + +-- System logs table +CREATE TABLE IF NOT EXISTS system_logs ( + id SERIAL PRIMARY KEY, + level VARCHAR(20), + source VARCHAR(100), + message TEXT, + created_at TIMESTAMP DEFAULT NOW() +); + +-- Statistics table +CREATE TABLE IF NOT EXISTS statistics ( + id SERIAL PRIMARY KEY, + metric_name VARCHAR(100), + metric_value DOUBLE PRECISION, + tags JSONB, + created_at TIMESTAMP DEFAULT NOW() +); + +-- Indexes +CREATE INDEX IF NOT EXISTS idx_conversations_device_session ON conversations(device_id, session_id); +CREATE INDEX IF NOT EXISTS idx_conversations_created_at ON conversations(created_at DESC); +CREATE INDEX IF NOT EXISTS idx_devices_status ON devices(status) WHERE status = 'online'; +CREATE INDEX IF NOT EXISTS idx_system_logs_level_created ON system_logs(level, created_at DESC); +VACUUM ANALYZE; +SQL +" + +# Step 7: Setup directories +print_status "[7/10] Setting up directories..." +ssh -p $SSH_PORT $SERVER "mkdir -p /opt/moltbot-monitoring /opt/moltbot-sync /opt/moltbot-backup/{database,sessions,disaster-recovery}" + +# Step 8: Copy monitoring configuration +print_status "[8/10] Setting up monitoring stack..." +ssh -p $SSH_PORT $SERVER "cd /opt/moltbot-monitoring && cat > docker-compose.yml << 'YAML' +version: '2.3' +services: + prometheus: + image: prom/prometheus:latest + container_name: moltbot-prometheus + command: + - '--config.file=/etc/prometheus/prometheus.yml' + - '--storage.tsdb.path=/prometheus' + ports: + - \"9090:9090\" + volumes: + - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml + - prometheus-data:/prometheus + restart: unless-stopped + + grafana: + image: grafana/grafana:latest + container_name: moltbot-grafana + ports: + - \"3000:3000\" + environment: + - GF_SECURITY_ADMIN_USER=admin + - GF_SECURITY_ADMIN_PASSWORD=moltbot2024 + - GF_USERS_ALLOW_SIGN_UP=false + volumes: + - grafana-data:/var/lib/grafana + restart: unless-stopped + + node-exporter: + image: prom/node-exporter:latest + container_name: moltbot-node-exporter + ports: + - \"9100:9100\" + restart: unless-stopped + +volumes: + prometheus-data: + grafana-data: +YAML + +mkdir -p prometheus +cat > prometheus/prometheus.yml << 'YAML' +global: + scrape_interval: 15s + +scrape_configs: + - job_name: 'prometheus' + static_configs: + - targets: ['localhost:9090'] + + - job_name: 'node-exporter' + static_configs: + - targets: ['node-exporter:9100'] +YAML +" + +# Step 9: Start services +print_status "[9/10] Starting services..." +ssh -p $SSH_PORT $SERVER "cd /opt/moltbot-monitoring && docker-compose up -d" + +# Step 10: Setup automated tasks +print_status "[10/10] Setting up automation..." +ssh -p $SSH_PORT $SERVER "cat > /etc/cron.d/moltbot-auto << 'CRON' +# Moltbot Automated Tasks +*/5 * * * * root curl -s http://localhost:18800/api/health > /dev/null +0 2 * * * root /opt/moltbot-backup/backup.sh +CRON +" + +# Summary +echo "" +print_status "========================================" +print_status " Deployment Complete!" +print_status "========================================" +echo "" +echo "Server: $SERVER" +echo "" +echo "Services deployed:" +echo " ✓ PostgreSQL (5432)" +echo " ✓ Prometheus (9090)" +echo " ✓ Grafana (3000) - admin/moltbot2024" +echo " ✓ Node Exporter (9100)" +echo "" +echo "Next steps:" +echo " 1. SSH to server: ssh -p $SSH_PORT $SERVER" +echo " 2. Configure Moltbot: cd /opt/moltbot" +echo " 3. Start Gateway: npm start" +echo "" +echo "For full configuration guide, see:" +echo " https://github.com/flowerjunjie/moltbot" +echo "" diff --git a/deploy-oneclick.sh b/deploy-oneclick.sh new file mode 100644 index 000000000..28beebb68 --- /dev/null +++ b/deploy-oneclick.sh @@ -0,0 +1,492 @@ +#!/bin/bash +# +# Moltbot One-Click Deployment +# Run this script on a fresh server to deploy complete Moltbot stack +# + +set -e + +# Colors +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' + +print_header() { + echo -e "${BLUE}========================================${NC}" + echo -e "${BLUE} $1${NC}" + echo -e "${BLUE}========================================${NC}" + echo "" +} + +print_info() { + echo -e "${GREEN}[INFO]${NC} $1" +} + +print_error() { + echo -e "${RED}[ERROR]${NC} $1" +} + +print_warning() { + echo -e "${YELLOW}[WARNING]${NC} $1" +} + +# Check if running as root +if [ "$EUID" -ne 0 ]; then + print_error "This script must be run as root" + print_info "Please run: sudo $0" + exit 1 +fi + +print_header "Moltbot One-Click Deployment v2.1" + +# Display system info +print_info "System Information" +echo " Hostname: $(hostname)" +echo " OS: $(lsb_release -d | cut -f2)" +echo " CPUs: $(nproc)" +echo " Memory: $(free -h | grep Mem | awk '{print $2}')" +echo " Disk: $(df -h / | tail -1 | awk '{print $4}') available" +echo "" + +# Confirm deployment +read -p "Continue with deployment? (yes/no): " confirm +if [ "$confirm" != "yes" ]; then + print_info "Deployment cancelled" + exit 0 +fi + +echo "" + +# Step 1: Update system +print_header "Step 1/12: Updating System" +apt-get update -qq +apt-get upgrade -y -qq +print_info "System updated" + +# Step 2: Install dependencies +print_header "Step 2/12: Installing Dependencies" +DEBIAN_FRONTEND=noninteractive apt-get install -y \ + curl \ + wget \ + git \ + python3 \ + python3-pip \ + postgresql \ + postgresql-contrib \ + nginx \ + docker.io \ + docker-compose \ + nodejs \ + npm \ + build-essential \ + iptables-persistent \ + keepalived \ + htop \ + vim \ + ufw + +print_info "Dependencies installed" + +# Step 3: Setup Docker +print_header "Step 3/12: Setting Up Docker" +systemctl start docker +systemctl enable docker +usermod -aG docker $SUDO_USER +print_info "Docker configured" + +# Step 4: Clone repository +print_header "Step 4/12: Cloning Moltbot Repository" +cd /opt +rm -rf moltbot +git clone https://github.com/flowerjunjie/moltbot.git moltbot +cd moltbot +print_info "Repository cloned" + +# Step 5: Install Python packages +print_header "Step 5/12: Installing Python Packages" +pip3 install -q psycopg2-binary psutil +print_info "Python packages installed" + +# Step 6: Setup PostgreSQL +print_header "Step 6/12: Setting Up PostgreSQL" +systemctl start postgresql +systemctl enable postgresql + +sudo -u postgres psql -c "CREATE DATABASE moltbot;" +sudo -u postgres psql -c "CREATE USER root WITH SUPERUSER;" +sudo -u postgres psql -c "ALTER USER root WITH PASSWORD '';" + +# Create tables +sudo -u postgres psql -d moltbot << 'SQL' +CREATE TABLE IF NOT EXISTS conversations ( + id SERIAL PRIMARY KEY, + device_id VARCHAR(100) NOT NULL, + session_id VARCHAR(100) NOT NULL, + role VARCHAR(20) NOT NULL, + content TEXT NOT NULL, + model VARCHAR(100), + tokens INTEGER, + created_at TIMESTAMP DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS devices ( + device_name VARCHAR(100) UNIQUE NOT NULL, + device_type VARCHAR(50), + ip_address VARCHAR(50), + last_seen TIMESTAMP DEFAULT NOW(), + status VARCHAR(20) DEFAULT 'online' +); + +CREATE TABLE IF NOT EXISTS system_logs ( + id SERIAL PRIMARY KEY, + level VARCHAR(20), + source VARCHAR(100), + message TEXT, + created_at TIMESTAMP DEFAULT NOW() +); + +CREATE TABLE IF NOT EXISTS statistics ( + id SERIAL PRIMARY KEY, + metric_name VARCHAR(100), + metric_value DOUBLE PRECISION, + tags JSONB, + created_at TIMESTAMP DEFAULT NOW() +); + +CREATE INDEX IF NOT EXISTS idx_conversations_device_session ON conversations(device_id, session_id); +CREATE INDEX IF NOT EXISTS idx_conversations_created_at ON conversations(created_at DESC); +CREATE INDEX IF NOT EXISTS idx_devices_status ON devices(status) WHERE status = 'online'; +CREATE INDEX IF NOT EXISTS idx_system_logs_level_created ON system_logs(level, created_at DESC); +VACUUM ANALYZE; +SQL + +print_info "PostgreSQL configured" + +# Step 7: Setup directories +print_header "Step 7/12: Setting Up Directories" +mkdir -p /opt/moltbot-monitoring/{prometheus,grafana/provisioning/datasources,grafana/provisioning/dashboards} +mkdir -p /opt/moltbot-sync +mkdir -p /opt/moltbot-backup/{database,sessions,disaster-recovery} +mkdir -p /var/log/moltbot +print_info "Directories created" + +# Step 8: Setup monitoring stack +print_header "Step 8/12: Setting Up Monitoring Stack" + +# Prometheus config +cat > /opt/moltbot-monitoring/prometheus/prometheus.yml << 'YAML' +global: + scrape_interval: 15s + +scrape_configs: + - job_name: 'prometheus' + static_configs: + - targets: ['localhost:9090'] + + - job_name: 'node-exporter' + static_configs: + - targets: ['node-exporter:9100'] + + - job_name: 'moltbot-metrics' + static_configs: + - targets: ['host.docker.internal:9101'] + scrape_interval: 10s +YAML + +# Grafana datasource +cat > /opt/moltbot-monitoring/grafana/provisioning/datasources/prometheus.yml << 'YAML' +apiVersion: 1 + +datasources: + - name: Prometheus + type: prometheus + access: proxy + url: http://prometheus:9090 + isDefault: true + editable: true +YAML + +# Docker Compose +cat > /opt/moltbot-monitoring/docker-compose.yml << 'YAML' +version: '2.3' + +services: + prometheus: + image: prom/prometheus:latest + container_name: moltbot-prometheus + command: + - '--config.file=/etc/prometheus/prometheus.yml' + - '--storage.tsdb.path=/prometheus' + ports: + - "9090:9090" + volumes: + - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml + - prometheus-data:/prometheus + restart: unless-stopped + + grafana: + image: grafana/grafana:latest + container_name: moltbot-grafana + ports: + - "3000:3000" + environment: + - GF_SECURITY_ADMIN_USER=admin + - GF_SECURITY_ADMIN_PASSWORD=moltbot2024 + - GF_USERS_ALLOW_SIGN_UP=false + volumes: + - grafana-data:/var/lib/grafana + - ./grafana/provisioning:/etc/grafana/provisioning + restart: unless-stopped + + node-exporter: + image: prom/node-exporter:latest + container_name: moltbot-node-exporter + ports: + - "9100:9100" + command: + - '--path.procfs=/host/proc' + - '--path.sysfs=/host/sys' + - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)' + volumes: + - /proc:/host/proc:ro + - /sys:/host/sys:ro + - /:/host:ro,rslave + restart: unless-stopped + +volumes: + prometheus-data: + grafana-data: +YAML + +cd /opt/moltbot-monitoring +docker-compose up -d +print_info "Monitoring stack started" + +# Step 9: Setup database API +print_header "Step 9/12: Setting Up Database API" + +cat > /opt/moltbot-sync/db-api.py << 'PYTHON' +#!/usr/bin/env python3 +# Database API for Moltbot + +import os +import sys +import json +import subprocess +from http.server import HTTPServer, BaseHTTPRequestHandler +from urllib.parse import urlparse, parse_qs +import psycopg2 + +DB_CONFIG = {'host': '/var/run/postgresql', 'database': 'moltbot', 'user': 'root'} + +def get_connection(): + return psycopg2.connect(**DB_CONFIG) + +class APIHandler(BaseHTTPRequestHandler): + def log_message(self, format, *args): pass + + def send_json(self, data, status=200): + self.send_response(status) + self.send_header('Content-Type', 'application/json') + self.send_header('Access-Control-Allow-Origin', '*') + self.end_headers() + self.wfile.write(json.dumps(data).encode()) + + def do_GET(self): + parsed = urlparse(self.path) + if parsed.path == '/api/health': + try: + conn = get_connection() + conn.close() + self.send_json({'status': 'healthy', 'database': 'connected'}) + except: + self.send_json({'status': 'unhealthy', 'database': 'disconnected'}, 503) + elif parsed.path == '/api/devices': + conn = get_connection() + cur = conn.cursor() + cur.execute('SELECT * FROM devices') + self.send_json({'devices': [dict(zip(['name', 'type', 'ip', 'last_seen', 'status'], row)) for row in cur.fetchall()]}) + conn.close() + else: + self.send_json({'error': 'Not found'}, 404) + +if __name__ == '__main__': + server = HTTPServer(('0.0.0.0', 18800), APIHandler) + print('Database API running on port 18800') + server.serve_forever() +PYTHON + +chmod +x /opt/moltbot-sync/db-api.py + +# Create systemd service +cat > /etc/systemd/system/moltbot-db-api.service << 'SERVICE' +[Unit] +Description=Moltbot Database API +After=network.target postgresql.service + +[Service] +Type=simple +User=root +WorkingDirectory=/opt/moltbot-sync +ExecStart=/usr/bin/python3 /opt/moltbot-sync/db-api.py +Restart=always + +[Install] +WantedBy=multi-user.target +SERVICE + +systemctl daemon-reload +systemctl enable moltbot-db-api +systemctl start moltbot-db-api +print_info "Database API started on port 18800" + +# Step 10: Setup metrics exporter +print_header "Step 10/12: Setting Up Metrics Exporter" + +cat > /usr/local/bin/moltbot-metrics.py << 'PYTHON' +#!/usr/bin/env python3 +import os +import psycopg2 +from http.server import HTTPServer, BaseHTTPRequestHandler + +DB_CONFIG = {'host': '/var/run/postgresql', 'database': 'moltbot', 'user': 'root'} + +class MetricsHandler(BaseHTTPRequestHandler): + def log_message(self, format, *args): pass + + def do_GET(self): + try: + conn = psycopg2.connect(**DB_CONFIG) + cur = conn.cursor() + cur.execute('SELECT COUNT(*) FROM devices WHERE status = %s', ('online',)) + online = cur.fetchone()[0] + cur.execute('SELECT COUNT(*) FROM devices') + total = cur.fetchone()[0] + conn.close() + + metrics = f'''# HELP moltbot_online_devices Number of online devices +# TYPE moltbot_online_devices gauge +moltbot_online_devices {online} +# HELP moltbot_total_devices Total number of devices +# TYPE moltbot_total_devices gauge +moltbot_total_devices {total}''' + + self.send_response(200) + self.send_header('Content-Type', 'text/plain') + self.end_headers() + self.wfile.write(metrics.encode()) + except Exception as e: + self.send_response(500) + self.end_headers() + +HTTPServer(('0.0.0.0', 9101), MetricsHandler).serve_forever() +PYTHON + +chmod +x /usr/local/bin/moltbot-metrics.py + +# Create systemd service +cat > /etc/systemd/system/moltbot-metrics.service << 'SERVICE' +[Unit] +Description=Moltbot Metrics Exporter +After=network.target postgresql.service + +[Service] +Type=simple +User=root +ExecStart=/usr/bin/python3 /usr/local/bin/moltbot-metrics.py +Restart=always + +[Install] +WantedBy=multi-user.target +SERVICE + +systemctl daemon-reload +systemctl enable moltbot-metrics +systemctl start moltbot-metrics +print_info "Metrics exporter started on port 9101" + +# Step 11: Setup automation +print_header "Step 11/12: Setting Up Automation" + +# Backup script +cat > /usr/local/bin/moltbot-backup-auto.sh << 'SCRIPT' +#!/bin/bash +DATE=$(date +%Y%m%d_%H%M%S) +pg_dump -U root moltbot | gzip > /opt/moltbot-backup/database/moltbot_$DATE.sql.gz +find /opt/moltbot-backup/database -name "*.sql.gz" -mtime -7 -delete +echo "Backup completed: $DATE" +SCRIPT + +chmod +x /usr/local/bin/moltbot-backup-auto.sh + +# Cron jobs +cat > /etc/cron.d/moltbot-auto << 'CRON' +# Moltbot Automation +*/5 * * * * root curl -s http://localhost:18800/api/health > /dev/null +0 2 * * * root /usr/local/bin/moltbot-backup-auto.sh +*/10 * * * * root /opt/moltbot-sync/sync-sessions.sh sync 2>/dev/null || true +CRON + +print_info "Automation configured" + +# Step 12: Setup firewall +print_header "Step 12/12: Setting Up Firewall" + +cat > /etc/iptables.rules << 'RULES' +*filter +:INPUT DROP [0:0] +:FORWARD DROP [0:0] +:OUTPUT ACCEPT [0:0] + +-A INPUT -i lo -j ACCEPT +-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT +-A INPUT -p tcp --dport 22 -j ACCEPT +-A INPUT -p tcp --dport 80 -j ACCEPT +-A INPUT -p tcp --dport 443 -j ACCEPT +-A INPUT -s 192.168.0.0/16 -p tcp --dport 18789 -j ACCEPT +-A INPUT -s 10.0.0.0/8 -p tcp --dport 18789 -j ACCEPT +-A INPUT -s 192.168.0.0/16 -p tcp --dport 18800 -j ACCEPT +-A INPUT -s 10.0.0.0/8 -p tcp --dport 18800 -j ACCEPT +-A INPUT -p tcp --dport 3000 -j ACCEPT +-A INPUT -p tcp --dport 9090 -j ACCEPT +-A INPUT -p tcp --dport 9100 -j ACCEPT +-A INPUT -p tcp --dport 9101 -j ACCEPT +-A INPUT -p icmp --icmp-type echo-request -j ACCEPT +COMMIT +RULES + +iptables-restore < /etc/iptables.rules +netfilter-persistent save +print_info "Firewall configured" + +# Final summary +echo "" +print_header "Deployment Complete!" +echo "" +print_info "Services Status:" +echo " ✓ PostgreSQL (5432)" +echo " ✓ Database API (18800)" +echo " ✓ Prometheus (9090)" +echo " ✓ Grafana (3000) - admin/moltbot2024" +echo " ✓ Node Exporter (9100)" +echo " ✓ Metrics Exporter (9101)" +echo "" +print_info "Access URLs:" +echo " Grafana: http://$(hostname -I | cut -d' ' -f1):3000" +echo " Prometheus: http://$(hostname -I | cut -d' ' -f1):9090" +echo " Database API: http://$(hostname -I | cut -d' ' -f1):18800" +echo "" +print_info "Quick Commands:" +echo " View logs: journalctl -u moltbot-db-api -f" +echo " Check status: systemctl status moltbot-*" +echo " Run backup: /usr/local/bin/moltbot-backup-auto.sh" +echo "" +print_info "Configuration files:" +echo " Database config: /opt/moltbot-sync/" +echo " Monitoring: /opt/moltbot-monitoring/" +echo " Backups: /opt/moltbot-backup/" +echo "" +echo -e "${GREEN}Moltbot is now ready!${NC}" +echo "" diff --git a/docker-compose-full.yml b/docker-compose-full.yml new file mode 100644 index 000000000..8ff037795 --- /dev/null +++ b/docker-compose-full.yml @@ -0,0 +1,191 @@ +version: '3.8' + +services: + # Moltbot Gateway + moltbot-gateway: + build: + context: . + dockerfile: Dockerfile.gateway + container_name: moltbot-gateway + ports: + - "18789:18789" + environment: + - NODE_ENV=production + - GATEWAY_MODE=hybrid + - GATEWAY_BIND=0.0.0.0 + - GATEWAY_AUTH_TOKEN=moltbot-cluster-2024 + volumes: + - moltbot-sessions:/root/.clawdbot/agents/main/sessions + - moltbot-config:/root/.clawdbot + depends_on: + - postgres + - redis + restart: unless-stopped + networks: + - moltbot-network + + # Database API + moltbot-db-api: + build: + context: . + dockerfile: Dockerfile.db-api + container_name: moltbot-db-api + ports: + - "18800:18800" + environment: + - DB_HOST=postgres + - DB_PORT=5432 + - DB_NAME=moltbot + - DB_USER=root + - DB_PASSWORD= + depends_on: + - postgres + restart: unless-stopped + networks: + - moltbot-network + + # PostgreSQL Database + postgres: + image: postgres:14-alpine + container_name: moltbot-postgres + ports: + - "5432:5432" + environment: + - POSTGRES_DB=moltbot + - POSTGRES_USER=root + - POSTGRES_HOST_AUTH_METHOD=trust + volumes: + - postgres-data:/var/lib/postgresql/data + - ./init-db.sql:/docker-entrypoint-initdb.d/init-db.sql + restart: unless-stopped + networks: + - moltbot-network + + # Redis (optional, for future use) + redis: + image: redis:7-alpine + container_name: moltbot-redis + ports: + - "6379:6379" + command: redis-server --appendonly yes + volumes: + - redis-data:/data + restart: unless-stopped + networks: + - moltbot-network + + # Prometheus Metrics + prometheus: + image: prom/prometheus:latest + container_name: moltbot-prometheus + ports: + - "9090:9090" + volumes: + - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml + - prometheus-data:/prometheus + command: + - '--config.file=/etc/prometheus/prometheus.yml' + - '--storage.tsdb.path=/prometheus' + restart: unless-stopped + networks: + - moltbot-network + + # Grafana Visualization + grafana: + image: grafana/grafana:latest + container_name: moltbot-grafana + ports: + - "3000:3000" + environment: + - GF_SECURITY_ADMIN_USER=admin + - GF_SECURITY_ADMIN_PASSWORD=moltbot2024 + - GF_USERS_ALLOW_SIGN_UP=false + - GF_INSTALL_PLUGINS=grafana-piechart-panel + volumes: + - grafana-data:/var/lib/grafana + - ./grafana/provisioning:/etc/grafana/provisioning + restart: unless-stopped + networks: + - moltbot-network + + # Node Exporter (system metrics) + node-exporter: + image: prom/node-exporter:latest + container_name: moltbot-node-exporter + ports: + - "9100:9100" + command: + - '--path.procfs=/host/proc' + - '--path.sysfs=/host/sys' + - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)' + volumes: + - /proc:/host/proc:ro + - /sys:/host/sys:ro + - /:/host:ro,rslave + restart: unless-stopped + networks: + - moltbot-network + + # Moltbot Metrics Exporter + metrics-exporter: + build: + context: . + dockerfile: Dockerfile.metrics + container_name: moltbot-metrics-exporter + ports: + - "9101:9101" + environment: + - DB_HOST=postgres + - DB_PORT=5432 + - DB_NAME=moltbot + - DB_USER=root + depends_on: + - postgres + restart: unless-stopped + networks: + - moltbot-network + + # Log Analyzer + log-analyzer: + build: + context: . + dockerfile: Dockerfile.log-analyzer + container_name: moltbot-log-analyzer + ports: + - "9102:9102" + volumes: + - /var/log:/var/log/host:ro + - ./logs:/app/logs + restart: unless-stopped + networks: + - moltbot-network + + # Nginx Reverse Proxy + nginx: + image: nginx:alpine + container_name: moltbot-nginx + ports: + - "80:80" + - "443:443" + volumes: + - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro + - ./nginx/ssl:/etc/nginx/ssl:ro + depends_on: + - moltbot-gateway + - grafana + - prometheus + restart: unless-stopped + networks: + - moltbot-network + +volumes: + postgres-data: + redis-data: + prometheus-data: + grafana-data: + moltbot-sessions: + moltbot-config: + +networks: + moltbot-network: + driver: bridge diff --git a/ha-setup.sh b/ha-setup.sh new file mode 100644 index 000000000..034e317f7 --- /dev/null +++ b/ha-setup.sh @@ -0,0 +1,337 @@ +#!/bin/bash +# +# Moltbot High Availability (HA) Configuration +# Configures redundant services and automatic failover +# + +set -e + +SERVER="root@38.14.254.51" + +echo "========================================" +echo " Moltbot High Availability Setup" +echo "========================================" +echo "" + +# Function to check if command exists +command_exists() { + command -v "$1" >/dev/null 2>&1 +} + +# Step 1: Install keepalived for VIP management +echo "[1/6] Installing keepalived for Virtual IP..." +ssh $SERVER "apt-get install -y keepalived" + +# Configure keepalived +ssh $SERVER "cat > /etc/keepalived/keepalived.conf << 'EOF' +vrrp_script chk_moltbot_gateway { + script \"curl -f http://localhost:18789/health || exit 1\" + interval 2 + weight 2 +} + +vrrp_instance VI_MOLTBOT { + state MASTER + interface eth0 + virtual_router_id 51 + priority 100 + advert_int 1 + + authentication { + auth_type PASS + auth_pass moltbot2024 + } + + virtual_ipaddress { + 38.14.254.100/24 + } + + track_script { + chk_moltbot_gateway + } + + notify_master \"/usr/local/bin/ha_notify.sh master\" + notify_backup \"/usr/local/bin/ha_notify.sh backup\" + notify_fault \"/usr/local/bin/ha_notify.sh fault\" +} +EOF +" + +echo "Keepalived configured" + +# Step 2: Create HA notification script +echo "[2/6] Creating HA notification script..." +ssh $SERVER "cat > /usr/local/bin/ha_notify.sh << 'SCRIPT' +#!/bin/bash +# HA State Change Notification + +STATE=\$1 +TIMESTAMP=\$(date +%Y%m%d_%H%M%S) +LOG=/var/log/moltbot-ha.log + +echo \"[\$TIMESTAMP] HA State changed to: \$STATE\" >> \$LOG + +case \$STATE in + master) + # Promote to master - start all services + systemctl start moltbot-gateway 2>/dev/null || true + systemctl start moltbot-db-api 2>/dev/null || true + echo \"This node is now MASTER\" | logger -t moltbot-ha + ;; + backup) + # Demote to backup - keep services running but ready + echo \"This node is now BACKUP\" | logger -t moltbot-ha + ;; + fault) + # Fault state - alert and try to recover + echo \"FAULT detected - attempting recovery\" | logger -t moltbot-ha -p error + systemctl restart moltbot-gateway 2>/dev/null || true + ;; +esac +SCRIPT +chmod +x /usr/local/bin/ha_notify.sh +" + +echo "HA notification script created" + +# Step 3: Setup PostgreSQL replication +echo "[3/6] Configuring PostgreSQL streaming replication..." +ssh $SERVER "cat > /etc/postgresql/14/main/conf.d/replication.conf << 'SQL' +# WAL Settings for Replication +wal_level = replica +max_wal_senders = 5 +max_replication_slots = 5 +hot_standby = on + +# Replication Slots +wal_keep_size = 1GB +SQL + +# Create replication user +psql -d moltbot -c \"CREATE USER replicator WITH REPLICATION ENCRYPTED PASSWORD 'replicator_pass';\" +psql -d moltbot -c \"ALTER USER replicator WITH REPLICATION;\" +" + +echo "PostgreSQL replication configured" + +# Step 4: Create automated failover script +echo "[4/6] Creating failover automation..." +ssh $SERVER "cat > /usr/local/bin/moltbot-failover.sh << 'SCRIPT' +#!/bin/bash +# Automated Failover Script + +GATEWAY_HEALTH_URL='http://localhost:18789/health' +DB_API_HEALTH_URL='http://localhost:18800/api/health' +CHECK_INTERVAL=10 +FAIL_THRESHOLD=3 +fail_count=0 + +log_message() { + echo \"[\$(date '+%Y-%m-%d %H:%M:%S')] \$1\" | tee -a /var/log/moltbot-failover.log +} + +check_service() { + local url=\$1 + local name=\$2 + + if curl -sf \"\$url\" > /dev/null 2>&1; then + log_message \"\$name is healthy\" + return 0 + else + log_message \"WARNING: \$name health check failed\" + return 1 + fi +} + +restart_service() { + local service=\$1 + log_message \"Attempting to restart \$service...\" + systemctl restart \$service + sleep 5 + + if systemctl is-active --quiet \$service; then + log_message \"\$service restarted successfully\" + return 0 + else + log_message \"ERROR: Failed to restart \$service\" + return 1 + fi +} + +# Main monitoring loop +log_message \"Failover monitor started\" + +while true; do + gateway_ok=true + db_api_ok=true + + # Check Gateway + if ! check_service \"\$GATEWAY_HEALTH_URL\" \"Gateway\"; then + gateway_ok=false + fi + + # Check Database API + if ! check_service \"\$DB_API_HEALTH_URL\" \"Database API\"; then + db_api_ok=false + fi + + # Handle failures + if [ \"\$gateway_ok\" = false ] || [ \"\$db_api_ok\" = false ]; then + fail_count=\$((fail_count + 1)) + log_message \"Fail count: \$fail_count/\$FAIL_THRESHOLD\" + + if [ \$fail_count -ge \$FAIL_THRESHOLD ]; then + log_message \"CRITICAL: Threshold reached, initiating recovery\" + + if [ \"\$gateway_ok\" = false ]; then + restart_service moltbot-gateway + fi + + if [ \"\$db_api_ok\" = false ]; then + restart_service moltbot-db-api + fi + + # Check database + if ! sudo -u postgres psql -c 'SELECT 1' >/dev/null 2>&1; then + log_message \"PostgreSQL not responding, restarting...\" + systemctl restart postgresql + fi + + fail_count=0 + fi + else + fail_count=0 + fi + + sleep \$CHECK_INTERVAL +done +SCRIPT +chmod +x /usr/local/bin/moltbot-failover.sh +" + +echo "Failover script created" + +# Step 5: Create systemd service for failover monitor +echo "[5/6] Creating failover monitor service..." +ssh $SERVER "cat > /etc/systemd/system/moltbot-failover.service << 'SERVICE' +[Unit] +Description=Moltbot Failover Monitor +After=network.target moltbot-gateway.service + +[Service] +Type=simple +ExecStart=/usr/local/bin/moltbot-failover.sh +Restart=always +RestartSec=10 +StandardOutput=journal +StandardError=journal + +[Install] +WantedBy=multi-user.target +SERVICE + +systemctl daemon-reload +systemctl enable moltbot-failover +systemctl start moltbot-failover +" + +echo "Failover monitor service started" + +# Step 6: Create disaster recovery backup +echo "[6/6] Creating disaster recovery backup..." +ssh $SERVER "cat > /usr/local/bin/moltbot-dr-backup.sh << 'SCRIPT' +#!/bin/bash +# Disaster Recovery Backup +# Creates complete system backup for DR purposes + +DR_BACKUP_DIR=\"/opt/moltbot-backup/disaster-recovery\" +DATE=\$(date +%Y%m%d_%H%M%S) +mkdir -p \"\$DR_BACKUP_DIR\" + +echo \"[\$(date)] Starting disaster recovery backup...\" + +# 1. Full database dump +echo \"Backing up PostgreSQL...\" +pg_dumpall -U root | gzip > \"\$DR_BACKUP_DIR/pg_all_\${DATE}.sql.gz\" + +# 2. Configuration files +echo \"Backing up configurations...\" +mkdir -p \"\$DR_BACKUP_DIR/config_\${DATE}\" +cp -r /root/.clawdbot/* \"\$DR_BACKUP_DIR/config_\${DATE}/\" 2>/dev/null || true +cp -r /opt/moltbot-monitoring/*.json \"\$DR_BACKUP_DIR/config_\${DATE}/\" 2>/dev/null || true +cp -r /etc/moltbot* \"\$DR_BACKUP_DIR/config_\${DATE}/\" 2>/dev/null || true + +# 3. Docker volumes +echo \"Backing up Docker volumes...\" +docker run --rm -v moltbot-monitoring_grafana-data:/data -v \"\$DR_BACKUP_DIR\":/backup busybox tar czf \"/backup/grafana_\${DATE}.tar.gz\" -C /data . +docker run --rm -v moltbot-monitoring_prometheus-data:/data -v \"\$DR_BACKUP_DIR\":/backup busybox tar czf \"/backup/prometheus_\${DATE}.tar.gz\" -C /data . + +# 4. System state +echo \"Capturing system state...\" +dpkg --get-selections > \"\$DR_BACKUP_DIR/packages_\${DATE}.list\" +iptables-save > \"\$DR_BACKUP_DIR/iptables_\${DATE}.rules\" + +# 5. Create recovery manifest +cat > \"\$DR_BACKUP_DIR/manifest_\${DATE}.txt\" << MANIFEST +Disaster Recovery Backup +Date: \$(date) +Hostname: \$(hostname) +IP Address: \$(hostname -I | cut -d' ' -f1) + +Contents: +- PostgreSQL full dump: pg_all_\${DATE}.sql.gz +- Configurations: config_\${DATE}/ +- Grafana data: grafana_\${DATE}.tar.gz +- Prometheus data: prometheus_\${DATE}.tar.gz +- Package list: packages_\${DATE}.list +- Firewall rules: iptables_\${DATE}.rules + +To restore: +1. Install PostgreSQL: apt-get install postgresql +2. Restore database: gunzip -c pg_all_\${DATE}.sql.gz | psql +3. Restore configs: cp -r config_\${DATE}/* / +4. Restore Docker: docker load < backups/*.tar +5. Restore packages: dpkg --set-selections < packages_\${DATE}.list +6. Restore firewall: iptables-restore < iptables_\${DATE}.rules +MANIFEST + +# 6. Cleanup old DR backups (keep last 3) +find \"\$DR_BACKUP_DIR\" -name \"pg_all_*.sql.gz\" -type f | sort -r | tail -n +4 | xargs rm -f +find \"\$DR_BACKUP_DIR\" -name \"config_*\" -type d | sort -r | tail -n +4 | xargs rm -rf + +# 7. Upload to remote storage (optional) +# You can add S3, rsync, or other remote backup here + +SIZE=\$(du -sh \"\$DR_BACKUP_DIR\" | cut -f1) +echo \"[\$(date)] DR backup completed. Size: \$SIZE\" +SCRIPT +chmod +x /usr/local/bin/moltbot-dr-backup.sh +" + +echo "Disaster recovery backup script created" + +# Summary +echo "" +echo "========================================" +echo " HA Configuration Complete!" +echo "========================================" +echo "" +echo "Configured Components:" +echo " ✓ Keepalived - Virtual IP (38.14.254.100)" +echo " ✓ HA notification script" +echo " ✓ PostgreSQL replication setup" +echo " ✓ Automated failover monitor" +echo " ✓ Disaster recovery backup" +echo "" +echo "Services:" +echo " moltbot-failover.service - Monitor & auto-recovery" +echo " keepalived.service - VIP management" +echo "" +echo "Commands:" +echo " /usr/local/bin/moltbot-failover.sh - Manual failover" +echo " /usr/local/bin/moltbot-dr-backup.sh - DR backup" +echo " systemctl status moltbot-failover - Check status" +echo "" +echo "Note: For full HA, deploy a secondary server with" +echo " priority 50 in keepalived.conf" +echo ""