Disaster Recovery 101: Architecting Auto-Failovers for MySQL and WordPress Deployments on OVH

Establishing a High-Availability MySQL Cluster with Replication and Orchestration

Achieving true disaster recovery for a WordPress deployment hinges on a robust, highly available MySQL backend. For production environments, a single MySQL instance is a single point of failure. We’ll architect a solution leveraging MySQL’s built-in replication and an orchestration layer to manage failover.

Our strategy involves setting up a primary-replica (master-slave) replication topology. The primary handles all write operations, while replicas asynchronously receive and apply changes. For automated failover, we’ll introduce a tool that monitors the primary’s health and promotes a replica if the primary becomes unavailable.

MySQL Replication Setup

This assumes you have at least two OVH cloud instances provisioned, each running MySQL. For simplicity, we’ll use two nodes: mysql-primary and mysql-replica. Ensure MySQL is installed and running on both.

1. Configure the Primary MySQL Server (mysql-primary):

Edit the MySQL configuration file (typically /etc/mysql/my.cnf or /etc/mysql/mysql.conf.d/mysqld.cnf).

[mysqld]
server-id = 1
log_bin = /var/log/mysql/mysql-bin.log
binlog_format = ROW
# Optional: For GTID-based replication, uncomment and configure
# gtid_mode = ON
# enforce_gtid_consistency = ON

Restart MySQL:

sudo systemctl restart mysql

2. Create a Replication User on the Primary:

-- Connect to MySQL as root
mysql -u root -p

-- Create a user for replication
CREATE USER 'repl_user'@'%' IDENTIFIED BY 'your_strong_password';
GRANT REPLICATION SLAVE ON *.* TO 'repl_user'@'%';
FLUSH PRIVILEGES;

-- Get the current binary log file and position
SHOW MASTER STATUS;
-- Note down the 'File' and 'Position' values. Example:
-- +------------------+----------+--------------+------------------+-------------------+
-- | File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
-- +------------------+----------+--------------+------------------+-------------------+
-- | mysql-bin.000001 |      154 |              |                  |                   |
-- +------------------+----------+--------------+------------------+-------------------+

3. Configure the Replica MySQL Server (mysql-replica):

Edit the MySQL configuration file on the replica.

[mysqld]
server-id = 2
relay_log = /var/log/mysql/mysql-relay-bin.log
read_only = 1
# Optional: For GTID-based replication
# gtid_mode = ON
# enforce_gtid_consistency = ON

Restart MySQL on the replica:

sudo systemctl restart mysql

4. Configure Replica to Connect to Primary:

-- Connect to MySQL as root on the replica
mysql -u root -p

-- Configure the replica to connect to the primary
-- Replace 'mysql-primary.yourdomain.com' with the actual IP or hostname
-- Replace 'your_strong_password' with the password for 'repl_user'
-- Use the 'File' and 'Position' noted from SHOW MASTER STATUS on the primary
CHANGE MASTER TO
  MASTER_HOST='mysql-primary.yourdomain.com',
  MASTER_USER='repl_user',
  MASTER_PASSWORD='your_strong_password',
  MASTER_LOG_FILE='mysql-bin.000001', -- From SHOW MASTER STATUS
  MASTER_LOG_POS=154;               -- From SHOW MASTER STATUS

-- Start the replication process
START SLAVE;

-- Check replication status
SHOW SLAVE STATUS\G
-- Look for 'Slave_IO_Running: Yes' and 'Slave_SQL_Running: Yes'.
-- Also, check 'Seconds_Behind_Master' which should ideally be 0 or very low.

Orchestrating Automated Failover with Orchestrator

MySQL’s built-in replication is asynchronous and lacks automated failover. We’ll use Orchestrator, a popular open-source tool for MySQL high availability and orchestration. It monitors replication topologies, detects failures, and can automatically promote replicas.

1. Install Orchestrator:

Orchestrator can be installed on a separate management node or on one of the MySQL nodes (though a separate node is recommended for better isolation).

Download the latest release from the Orchestrator GitHub repository and install it. For Debian/Ubuntu:

wget https://github.com/openark/orchestrator/releases/download/v3.2.7/orchestrator_3.2.7_linux_amd64.deb
sudo dpkg -i orchestrator_3.2.7_linux_amd64.deb

2. Configure Orchestrator:

Create a configuration file, e.g., /etc/orchestrator/orchestrator.conf.json.

{
  "Debug": false,
  "ListenAddress": ":3000",
  "MySQLTopologyUser": "orchestrator",
  "MySQLTopologyPassword": "your_orchestrator_db_password",
  "MySQLOrchestratorHostPort": "127.0.0.1:3306",
  "MySQLOrchestratorDatabaseName": "orchestrator",
  "DiscoveryPeriodSeconds": 10,
  "FailureDetectionPeriodSeconds": 10,
  "PromotionLagSeconds": 60,
  "RecoveryPeriodBlockSeconds": 3600,
  "DetectClusterTopologyOrder": "discover,detect-unseen,detect-orphan,detect-replication-unison,detect-duplicate-primary,detect-unhealthy",
  "AutoDiscoverAtStartup": true,
  "PostponePromotionOnFailureToDiscoverSeconds": 300,
  "SlaveLagQuery": "SELECT * FROM mysql.slave_master_info",
  "SlaveLagQueryForAllReplicas": true,
  "GlobalWriteableCells": [],
  "GlobalReadOnlyCells": [],
  "Cell": "default",
  "ClusterName": "wordpress_cluster",
  "SnapshotTopologies": true,
  "SnapshotTopologiesPeriodSeconds": 3600,
  "SnapshotTopologiesMaxCount": 10,
  "HTTPProxyórios": [
    {
      "Host": "localhost",
      "Port": 8080
    }
  ],
  "Hooks": {
    "PreFailoverProcesses": [],
    "PostFailoverProcesses": [],
    "PrePromotionProcesses": [],
    "PostPromotionProcesses": [],
    "PreGroomProcesses": [],
    "PostGroomProcesses": []
  },
  "KVStore": "orchestrator",
  "KVStoreCredentials": {
    "orchestrator": "your_kv_store_password"
  },
  "KVStoreBackend": "redis",
  "KVStoreRedisAddress": "localhost:6379",
  "KVStoreRedisDatabase": 0,
  "KVStoreRedisPassword": "",
  "KVStoreRedisKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterMode": false,
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisClusterDatabase": 0,
  "KVStoreRedisClusterKeyPrefix": "orchestrator:",
  "KVStoreRedisSentinelMode": false,
  "KVStoreRedisSentinelMasterName": "",
  "KVStoreRedisSentinelAddresses": [],
  "KVStoreRedisSentinelPassword": "",
  "KVStoreRedisSentinelDatabase": 0,
  "KVStoreRedisSentinelKeyPrefix": "orchestrator:",
  "KVStoreRedisClusterNodes": [],
  "KVStoreRedisClusterPassword": "",
  "KVStoreRedisCluster