Backup & Restore

A robust backup and restore strategy is essential for protecting your Prisme.ai platform data and ensuring business continuity. This guide provides detailed instructions for backing up and restoring all components of your self-hosted Prisme.ai environment.

Backup Strategy

Your Prisme.ai platform requires backing up several components:

Client-Managed Databases

MongoDB/compatible database
Elasticsearch/OpenSearch
Redis

Object Storage

S3 or compatible object storage
Document files and attachments

Configuration

Kubernetes manifests
Helm values
Terraform state files

Secrets

Kubernetes secrets
Certificate files
API keys and credentials

Your Prisme.ai platform requires backing up several components:

Client-Managed Databases

MongoDB/compatible database
Elasticsearch/OpenSearch
Redis

Object Storage

S3 or compatible object storage
Document files and attachments

Configuration

Kubernetes manifests
Helm values
Terraform state files

Secrets

Kubernetes secrets
Certificate files
API keys and credentials

Recommended backup frequencies based on data criticality:

Component	Frequency	Retention
MongoDB	Daily	30 days
Elasticsearch/OpenSearch	Daily	14 days
Redis	Daily	7 days
S3 Storage	Weekly (incremental)	90 days
Configuration	After changes	Last 10 versions

Adjust backup frequency based on your organization’s Recovery Point Objective (RPO) requirements.

Database Backup Procedures

Create MongoDB Backup

Use mongodump to create a full backup of your MongoDB database:

# For standalone MongoDB
mongodump --uri="mongodb://username:password@hostname:port/database" \
  --out=/path/to/backup/mongo/$(date +%Y-%m-%d)

# For MongoDB Atlas
mongodump --uri="mongodb+srv://username:password@cluster.mongodb.net/database" \
  --out=/path/to/backup/mongo/$(date +%Y-%m-%d)

MongoDB Backup Options

Additional options to consider:

# Compressed backup (reduces storage requirements)
mongodump --uri="mongodb://username:password@hostname:port/database" \
  --out=/path/to/backup/mongo/$(date +%Y-%m-%d) \
  --gzip

# Backup specific collections
mongodump --uri="mongodb://username:password@hostname:port/database" \
  --collection=users --collection=agents \
  --out=/path/to/backup/mongo/$(date +%Y-%m-%d)

Verify MongoDB Backup

Ensure the backup contains all expected data:

# List databases in the backup
find /path/to/backup/mongo/$(date +%Y-%m-%d) -type d -depth 1

# Count documents in a specific collection
mongorestore --uri="mongodb://username:password@hostname:port/database" \
  --nsInclude="database.collection" \
  --dryRun /path/to/backup/mongo/$(date +%Y-%m-%d) | grep "documents to restore"

Schedule Regular Backups

Create a cron job to automate daily backups:

# Add to crontab
0 1 * * * /path/to/mongodb-backup-script.sh > /path/to/logs/mongodb-backup-$(date +\%Y-\%m-\%d).log 2>&1

Example backup script (mongodb-backup-script.sh):

#!/bin/bash

BACKUP_DIR="/path/to/backup/mongo"
DATE=$(date +%Y-%m-%d)
RETENTION_DAYS=30

# Create backup
mongodump --uri="mongodb://username:password@hostname:port/database" \
  --out=${BACKUP_DIR}/${DATE} --gzip

# Clean up old backups
find ${BACKUP_DIR} -type d -mtime +${RETENTION_DAYS} -exec rm -rf {} \;

Create MongoDB Backup

Use mongodump to create a full backup of your MongoDB database:

# For standalone MongoDB
mongodump --uri="mongodb://username:password@hostname:port/database" \
  --out=/path/to/backup/mongo/$(date +%Y-%m-%d)

# For MongoDB Atlas
mongodump --uri="mongodb+srv://username:password@cluster.mongodb.net/database" \
  --out=/path/to/backup/mongo/$(date +%Y-%m-%d)

MongoDB Backup Options

Additional options to consider:

# Compressed backup (reduces storage requirements)
mongodump --uri="mongodb://username:password@hostname:port/database" \
  --out=/path/to/backup/mongo/$(date +%Y-%m-%d) \
  --gzip

# Backup specific collections
mongodump --uri="mongodb://username:password@hostname:port/database" \
  --collection=users --collection=agents \
  --out=/path/to/backup/mongo/$(date +%Y-%m-%d)

Verify MongoDB Backup

Ensure the backup contains all expected data:

# List databases in the backup
find /path/to/backup/mongo/$(date +%Y-%m-%d) -type d -depth 1

# Count documents in a specific collection
mongorestore --uri="mongodb://username:password@hostname:port/database" \
  --nsInclude="database.collection" \
  --dryRun /path/to/backup/mongo/$(date +%Y-%m-%d) | grep "documents to restore"

Schedule Regular Backups

Create a cron job to automate daily backups:

# Add to crontab
0 1 * * * /path/to/mongodb-backup-script.sh > /path/to/logs/mongodb-backup-$(date +\%Y-\%m-\%d).log 2>&1

Example backup script (mongodb-backup-script.sh):

#!/bin/bash

BACKUP_DIR="/path/to/backup/mongo"
DATE=$(date +%Y-%m-%d)
RETENTION_DAYS=30

# Create backup
mongodump --uri="mongodb://username:password@hostname:port/database" \
  --out=${BACKUP_DIR}/${DATE} --gzip

# Clean up old backups
find ${BACKUP_DIR} -type d -mtime +${RETENTION_DAYS} -exec rm -rf {} \;

Create Elasticsearch/OpenSearch Backup

Use Elasticsearch snapshot API or elasticdump for backups:

Using Snapshot API (recommended)

# 1. Register a snapshot repository (S3)
curl -X PUT "localhost:9200/_snapshot/backup_repository" -H "Content-Type: application/json" -d'
{
  "type": "s3",
  "settings": {
    "bucket": "your-backup-bucket",
    "region": "your-region",
    "role_arn": "arn:aws:iam::account-id:role/role-name"
  }
}'

# 2. Create a snapshot
curl -X PUT "localhost:9200/_snapshot/backup_repository/snapshot_$(date +%Y%m%d)" -H "Content-Type: application/json" -d'
{
  "indices": "*",
  "ignore_unavailable": true,
  "include_global_state": true
}'

# 3. Check snapshot status
curl -X GET "localhost:9200/_snapshot/backup_repository/snapshot_$(date +%Y%m%d)/_status"

Using elasticdump

# Install elasticdump
npm install -g elasticdump

# Backup all indices data
elasticdump \
  --input=http://elasticsearch:9200/ \
  --output=/path/to/backup/elasticsearch/data_$(date +%Y%m%d).json \
  --type=data

# Backup index mappings
elasticdump \
  --input=http://elasticsearch:9200/ \
  --output=/path/to/backup/elasticsearch/mapping_$(date +%Y%m%d).json \
  --type=mapping

Verify Elasticsearch/OpenSearch Backup

Ensure the backup contains all expected indices and data:

# For snapshot API
curl -X GET "localhost:9200/_snapshot/backup_repository/snapshot_$(date +%Y%m%d)"

# For elasticdump
jq 'keys' /path/to/backup/elasticsearch/mapping_$(date +%Y%m%d).json

Schedule Regular Backups

Create a cron job to automate daily backups:

# Add to crontab (for elasticdump method)
0 2 * * * /path/to/elasticsearch-backup-script.sh > /path/to/logs/elasticsearch-backup-$(date +\%Y-\%m-\%d).log 2>&1

Configure Redis Persistence

Ensure Redis is configured for persistence:

# In redis.conf

# RDB persistence
save 900 1
save 300 10
save 60 10000

# AOF persistence
appendonly yes
appendfsync everysec

Create Redis Backup

Use the SAVE command or copy the RDB file:

# Trigger a SAVE operation
redis-cli -h redis-host -a password SAVE

# Copy the dump.rdb file
cp /path/to/redis/data/dump.rdb /path/to/backup/redis/dump_$(date +%Y%m%d).rdb

Schedule Regular Backups

Create a cron job to automate daily backups:

# Add to crontab
0 3 * * * /path/to/redis-backup-script.sh > /path/to/logs/redis-backup-$(date +\%Y-\%m-\%d).log 2>&1

Back Up S3 Data

Use AWS CLI or compatible tools to back up your S3 storage:

# Sync to another bucket (cross-region for disaster recovery)
aws s3 sync s3://your-prisme-bucket s3://your-backup-bucket

# Or sync to local storage
aws s3 sync s3://your-prisme-bucket /path/to/backup/s3-data

Enable Versioning

Enable versioning on your S3 bucket for built-in backup protection:

aws s3api put-bucket-versioning \
  --bucket your-prisme-bucket \
  --versioning-configuration Status=Enabled

Set Up Lifecycle Policies

Configure lifecycle policies to manage backup retention:

aws s3api put-bucket-lifecycle-configuration \
  --bucket your-backup-bucket \
  --lifecycle-configuration file://lifecycle-config.json

Example lifecycle-config.json:

{
  "Rules": [
    {
      "ID": "Delete old backups",
      "Status": "Enabled",
      "Prefix": "",
      "Expiration": {
        "Days": 90
      }
    }
  ]
}

Configuration Backup

Back Up Kubernetes Resources

Save your Kubernetes configuration resources:

# Back up all resources in the Prisme namespace
mkdir -p /path/to/backup/kubernetes/$(date +%Y-%m-%d)

# Export deployments, services, configmaps, secrets, etc.
kubectl get all,pvc,cm,secret -n prisme-system -o yaml > \
  /path/to/backup/kubernetes/$(date +%Y-%m-%d)/prisme-resources.yaml

Back Up Helm Values

Save your Helm chart values for each release:

# Get values for each Helm release
helm get values prisme-core -n prisme-system -o yaml > \
  /path/to/backup/helm/$(date +%Y-%m-%d)/prisme-core-values.yaml

helm get values prisme-securechat -n prisme-system -o yaml > \
  /path/to/backup/helm/$(date +%Y-%m-%d)/prisme-securechat-values.yaml

# Repeat for each product module

Back Up Terraform State

If using Terraform, back up your state files:

# If using local state
cp terraform.tfstate terraform.tfstate.backup
cp -r terraform.tfstate.d /path/to/backup/terraform/$(date +%Y-%m-%d)

# If using remote state (recommended), ensure your remote backend has proper backup

Using remote state in Terraform (like S3 with versioning or Terraform Cloud) provides built-in backup capabilities.

Restore Procedures

Prepare for Restore

Before restoring, stop services that interact with the database:

# Scale down Prisme.ai deployments
kubectl scale deployment -n prisme-system --replicas=0 \
  prisme-api prisme-worker prisme-securechat prisme-knowledge

Restore MongoDB Data

Use mongorestore to restore from your backup:

# Full restore
mongorestore --uri="mongodb://username:password@hostname:port/database" \
  --nsFrom="database.*" --nsTo="database.*" \
  /path/to/backup/mongo/YYYY-MM-DD

# Or restore specific collections
mongorestore --uri="mongodb://username:password@hostname:port/database" \
  --nsInclude="database.users" --nsInclude="database.agents" \
  /path/to/backup/mongo/YYYY-MM-DD

Restoring will overwrite existing data. Be sure to validate your backup before proceeding.

Restart Services

After restore is complete, scale the services back up:

# Scale up Prisme.ai deployments
kubectl scale deployment -n prisme-system --replicas=1 \
  prisme-api prisme-worker prisme-securechat prisme-knowledge

# Verify pods are running
kubectl get pods -n prisme-system

Prepare for Restore

Before restoring, stop services that interact with the database:

# Scale down Prisme.ai deployments
kubectl scale deployment -n prisme-system --replicas=0 \
  prisme-api prisme-worker prisme-securechat prisme-knowledge

Restore MongoDB Data

Use mongorestore to restore from your backup:

# Full restore
mongorestore --uri="mongodb://username:password@hostname:port/database" \
  --nsFrom="database.*" --nsTo="database.*" \
  /path/to/backup/mongo/YYYY-MM-DD

# Or restore specific collections
mongorestore --uri="mongodb://username:password@hostname:port/database" \
  --nsInclude="database.users" --nsInclude="database.agents" \
  /path/to/backup/mongo/YYYY-MM-DD

Restoring will overwrite existing data. Be sure to validate your backup before proceeding.

Restart Services

After restore is complete, scale the services back up:

# Scale up Prisme.ai deployments
kubectl scale deployment -n prisme-system --replicas=1 \
  prisme-api prisme-worker prisme-securechat prisme-knowledge

# Verify pods are running
kubectl get pods -n prisme-system

Prepare for Restore

Stop services that interact with Elasticsearch:

kubectl scale deployment -n prisme-system --replicas=0 \
  prisme-api prisme-worker

Restore Data

Using Snapshot API

# List available snapshots
curl -X GET "localhost:9200/_snapshot/backup_repository/_all"

# Restore from snapshot
curl -X POST "localhost:9200/_snapshot/backup_repository/snapshot_YYYYMMDD/_restore" -H "Content-Type: application/json" -d'
{
  "indices": "*",
  "ignore_unavailable": true,
  "include_global_state": true
}'

# Check restore status
curl -X GET "localhost:9200/_recovery?human"

Using elasticdump

# Restore mappings first
elasticdump \
  --input=/path/to/backup/elasticsearch/mapping_YYYYMMDD.json \
  --output=http://elasticsearch:9200/ \
  --type=mapping

# Then restore data
elasticdump \
  --input=/path/to/backup/elasticsearch/data_YYYYMMDD.json \
  --output=http://elasticsearch:9200/ \
  --type=data

Restart Services

After restore is complete, restart the services:

kubectl scale deployment -n prisme-system --replicas=1 \
  prisme-api prisme-worker

# Verify pods are running
kubectl get pods -n prisme-system

Stop Redis Server

If possible, gracefully shut down Redis:

redis-cli -h redis-host -a password SHUTDOWN SAVE

If using Redis in Kubernetes:

# Scale down Redis deployment
kubectl scale statefulset -n prisme-system redis --replicas=0

Replace RDB File

Copy the backup RDB file to Redis data directory:

# If using persistent volume in Kubernetes
kubectl cp /path/to/backup/redis/dump_YYYYMMDD.rdb \
  prisme-system/redis-0:/data/dump.rdb

# For standalone Redis
cp /path/to/backup/redis/dump_YYYYMMDD.rdb /path/to/redis/data/dump.rdb

Restart Redis

Start Redis service with the restored data:

# For Kubernetes
kubectl scale statefulset -n prisme-system redis --replicas=1

# For standalone Redis
systemctl start redis

# Verify Redis is running with restored data
redis-cli -h redis-host -a password info keyspace

Prepare for Restore

Consider the impact of restoration on your application:

# Scale down services that interact with object storage
kubectl scale deployment -n prisme-system --replicas=0 \
  prisme-document-processor prisme-file-handler

Restore S3 Data

Sync data from your backup location:

# From backup bucket
aws s3 sync s3://your-backup-bucket s3://your-prisme-bucket

# Or from local backup
aws s3 sync /path/to/backup/s3-data s3://your-prisme-bucket

If using versioning, you can restore specific versions of objects:

# List versions of a specific object
aws s3api list-object-versions --bucket your-prisme-bucket --prefix path/to/object

# Restore specific version
aws s3api copy-object --copy-source your-prisme-bucket/path/to/object?versionId=VERSION_ID \
  --bucket your-prisme-bucket --key path/to/object

Verify Restoration

Check that files have been properly restored:

# List objects in bucket
aws s3 ls s3://your-prisme-bucket --recursive | head

# Count objects
aws s3 ls s3://your-prisme-bucket --recursive | wc -l

Restart Services

Resume normal operations:

kubectl scale deployment -n prisme-system --replicas=1 \
  prisme-document-processor prisme-file-handler

Configuration Restore

Restore Kubernetes Resources

Apply your backed-up Kubernetes configurations:

# First, clean up the namespace if necessary
# WARNING: This will delete all resources! Use with caution.
# kubectl delete namespace prisme-system
# kubectl create namespace prisme-system

# Apply backed-up resources
kubectl apply -f /path/to/backup/kubernetes/YYYY-MM-DD/prisme-resources.yaml

Be cautious when restoring resources. Consider restoring specific resource types instead of everything at once:

# Restore only ConfigMaps and Secrets first
kubectl apply -f /path/to/backup/kubernetes/YYYY-MM-DD/configmaps.yaml
kubectl apply -f /path/to/backup/kubernetes/YYYY-MM-DD/secrets.yaml

# Then restore other resources
kubectl apply -f /path/to/backup/kubernetes/YYYY-MM-DD/deployments.yaml

Restore Helm Releases

Use your backed-up values to reinstall or upgrade Helm releases:

# Reinstall core
helm upgrade --install prisme-core prisme/prisme-core \
  -f /path/to/backup/helm/YYYY-MM-DD/prisme-core-values.yaml \
  --namespace prisme-system

# Reinstall product modules
helm upgrade --install prisme-securechat prisme/prisme-securechat \
  -f /path/to/backup/helm/YYYY-MM-DD/prisme-securechat-values.yaml \
  --namespace prisme-system

# Repeat for other product modules

Restore Terraform State

If you need to restore Terraform state:

# For local state
cp /path/to/backup/terraform/YYYY-MM-DD/terraform.tfstate .
cp -r /path/to/backup/terraform/YYYY-MM-DD/terraform.tfstate.d .

# Verify state
terraform state list

For remote state, follow your backend provider’s restoration process.

Disaster Recovery Planning

Define your recovery objectives to guide your backup strategy:

RPO (Recovery Point Objective)

Maximum acceptable data loss in time:

Critical data: RPO < 1 hour
Important data: RPO < 24 hours
Regular data: RPO < 1 week

RTO (Recovery Time Objective)

Maximum acceptable time to restore service:

Critical services: RTO < 4 hours
Important services: RTO < 24 hours
Regular services: RTO < 3 days

Testing and Validation

Verify Backup Integrity

Regularly test your backups to ensure they can be restored:

# For MongoDB
mongorestore --uri="mongodb://username:password@hostname:port/test_db" \
  --nsFrom="database.*" --nsTo="test_db.*" \
  --dryRun /path/to/backup/mongo/YYYY-MM-DD

# For S3
aws s3 cp s3://your-backup-bucket/sample-file.pdf /tmp/test-restore.pdf

Validation Checkpoints

Establish validation points for successful restoration:

Data Validation

Record counts match pre-backup state
Sample record content is intact
Relationships between data are preserved
Application-specific data tests pass

Functionality Validation

Core services start successfully
API endpoints respond correctly
Authentication and authorization work
Data processing functions operate properly
UI elements display and function as expected

Document Restoration Procedures

Maintain detailed, tested restoration runbooks:

Step-by-step instructions
Required credentials and access
Validation checkpoints
Troubleshooting guidance
Contact information for support

Overview

Cloud Providers

Docker & Kubernetes Deployment

Entreprise Services

AI Products

Operations

Backup Strategy

Client-Managed Databases

Object Storage

Configuration

Secrets

Client-Managed Databases

Object Storage

Configuration

Secrets

Database Backup Procedures

Configuration Backup

Restore Procedures

Configuration Restore

Disaster Recovery Planning

RPO (Recovery Point Objective)

RTO (Recovery Time Objective)

RPO (Recovery Point Objective)

RTO (Recovery Time Objective)

Testing and Validation

Data Validation

Functionality Validation

Next Steps

Updates

Scaling

Overview

Cloud Providers

Docker & Kubernetes Deployment

Entreprise Services

AI Products

Operations

​Backup Strategy

Client-Managed Databases

Object Storage

Configuration

Secrets

Client-Managed Databases

Object Storage

Configuration

Secrets

​Database Backup Procedures

​Configuration Backup

​Restore Procedures

​Configuration Restore

​Disaster Recovery Planning

RPO (Recovery Point Objective)

RTO (Recovery Time Objective)

RPO (Recovery Point Objective)

RTO (Recovery Time Objective)

​Testing and Validation

Data Validation

Functionality Validation

​Next Steps

Updates

Scaling

Backup Strategy

Database Backup Procedures

Configuration Backup

Restore Procedures

Configuration Restore

Disaster Recovery Planning

Testing and Validation

Next Steps