ops: add integration secret rotation and offsite backup alerting

infra/deploy/OFFSITE_BACKUP_AND_ALERTING.md

@@ -0,0 +1,82 @@
+# Offsite Backup Replication + Failure Alerting
+
+This runbook configures:
+
+1. Encrypted local DB backups
+2. Replication to S3-compatible object storage (AWS S3, Backblaze B2 S3, Wasabi)
+3. Webhook/email alerting on backup or restore-test failure
+
+## 1) Configure secrets file
+
+Create or edit:
+
+`/opt/proxpanel/.backup.env`
+
+Reference template:
+
+`/opt/proxpanel/infra/deploy/.backup.env.example`
+
+## 2) Example provider configs
+
+### AWS S3
+
+```bash
+OFFSITE_BACKUP_ENABLED=true
+OFFSITE_S3_BUCKET=my-proxpanel-backups
+OFFSITE_S3_REGION=us-east-1
+OFFSITE_S3_PREFIX=proxpanel/db
+OFFSITE_S3_ACCESS_KEY_ID=AKIA...
+OFFSITE_S3_SECRET_ACCESS_KEY=...
+OFFSITE_S3_ENDPOINT_URL=
+```
+
+### Wasabi
+
+```bash
+OFFSITE_BACKUP_ENABLED=true
+OFFSITE_S3_BUCKET=my-proxpanel-backups
+OFFSITE_S3_REGION=us-east-1
+OFFSITE_S3_PREFIX=proxpanel/db
+OFFSITE_S3_ENDPOINT_URL=https://s3.us-east-1.wasabisys.com
+OFFSITE_S3_ACCESS_KEY_ID=...
+OFFSITE_S3_SECRET_ACCESS_KEY=...
+```
+
+### Backblaze B2 (S3 Compatible)
+
+```bash
+OFFSITE_BACKUP_ENABLED=true
+OFFSITE_S3_BUCKET=my-proxpanel-backups
+OFFSITE_S3_REGION=us-west-002
+OFFSITE_S3_PREFIX=proxpanel/db
+OFFSITE_S3_ENDPOINT_URL=https://s3.us-west-002.backblazeb2.com
+OFFSITE_S3_ACCESS_KEY_ID=...
+OFFSITE_S3_SECRET_ACCESS_KEY=...
+```
+
+## 3) Configure alerting
+
+Set one or both:
+
+```bash
+BACKUP_ALERT_WEBHOOK_URL=https://hooks.example.com/proxpanel-backup
+BACKUP_ALERT_EMAIL_WEBHOOK_URL=https://mailer.example.com/send
+BACKUP_ALERT_EMAIL_TO=ops@votcloud.com
+```
+
+## 4) Apply cron schedule
+
+```bash
+sudo bash /opt/proxpanel/infra/deploy/configure-db-backup-cron.sh --run-now
+```
+
+## 5) Verification
+
+1. Check local encrypted backup exists in `/opt/proxpanel-backups/daily/<timestamp>/`.
+2. Check offsite files:
+   - `proxpanel.sql.enc`
+   - `proxpanel.sql.enc.sha256`
+3. Check logs:
+   - `/var/log/proxpanel-db-backup.log`
+   - `/var/log/proxpanel-db-restore-test.log`
+4. Trigger controlled failure and confirm alert received (webhook/email).