Disaster Recovery 101: Architecting Auto-Failovers for MySQL and Perl Deployments on DigitalOcean
Establishing a High-Availability MySQL Cluster with Orchestrator
For critical applications, a single MySQL instance is a single point of failure. Architecting for high availability (HA) necessitates a robust failover strategy. We’ll leverage Orchestrator, a popular MySQL replication topology manager, to automate this process. Orchestrator monitors replication health and can automatically promote a replica to a primary in case of failure.
Our setup will involve a primary MySQL instance, one or more replicas, and a separate DigitalOcean Droplet dedicated to running Orchestrator. This dedicated Orchestrator instance will connect to all MySQL nodes and manage their replication relationships.
Orchestrator Installation and Configuration
First, provision a new DigitalOcean Droplet for Orchestrator. A small, inexpensive Droplet (e.g., 1 vCPU, 1GB RAM) is typically sufficient. Ensure it has network access to your MySQL Droplets.
Install Orchestrator on this new Droplet. We’ll use `wget` and `dpkg` for a Debian/Ubuntu system. The exact version might change, so consult the official Orchestrator GitHub repository for the latest stable release.
Installing Orchestrator (Debian/Ubuntu)
wget https://github.com/openark/orchestrator/releases/download/v3.2.7/orchestrator_3.2.7_linux_amd64.deb sudo dpkg -i orchestrator_3.2.7_linux_amd64.deb
Orchestrator Configuration File
The primary configuration file is located at /etc/orchestrator/orchestrator.conf.json. We need to configure database credentials for Orchestrator to connect to MySQL and define its behavior. Ensure the user specified has sufficient privileges (e.g., `REPLICATION SLAVE`, `REPLICATION CLIENT`, `SUPER`, `PROCESS`, `SELECT`).
{
"Debug": false,
"ListenAddress": ":3000",
"MySQLTopologyUser": "orchestrator",
"MySQLTopologyPassword": "your_orchestrator_db_password",
"MySQLOrchestratorHost": "127.0.0.1",
"MySQLOrchestratorPort": 3306,
"MySQLOrchestratorDatabase": "orchestrator",
"MySQLOrchestratorUser": "orchestrator",
"MySQLOrchestratorPassword": "your_orchestrator_db_password",
"SlaveLagQuery": "SELECT * FROM mysql.slave_lag_info WHERE SlaveHost = ? AND SlavePort = ?",
"PromotionUser": "orchestrator_promote",
"PromotionPassword": "your_promotion_password",
"PromotionForgetMasterSeconds": 3600,
"DiscoveryPeriodSeconds": 10,
"FailureDetectionPeriodSeconds": 5,
"FailureDetectionTimePeriodSeconds": 60,
"RecoveryPeriodSeconds": 10,
"DetectClusterAliasQuery": "SELECT @@global.server_id",
"DetectClusterNameQuery": "SELECT @@global.server_id",
"PreemptivePaging": true,
"PostponePromotionOnLagSeconds": 300,
"PostponeMaintenanceOnLagSeconds": 300,
"AutoOrchestrate": true,
"AutoDiscoverByClusterName": true,
"AutoDiscoverByClusterAlias": true,
"AutoDiscoverByHost": true,
"AutoDiscoverByMySQLHostName": true,
"AutoDiscoverByMySQLPort": true,
"AutoDiscoverByMySQLUser": true,
"AutoDiscoverByMySQLPassword": true,
"AutoDiscoverByMySQLSlaveLag": true,
"AutoDiscoverByMySQLSlaveIORunning": true,
"AutoDiscoverByMySQLSlaveSQLRunning": true,
"AutoDiscoverByMySQLSlaveSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterIORunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSQLRunning": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSecondsBehindMaster": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterHost": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterMasterPort": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterUser": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterPassword": true,
"AutoDiscoverByMySQLSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveMasterSlaveLag": true,
"AutoDiscoverByMySQLSlaveMaster