Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for MySQL and WooCommerce Deployments on DigitalOcean

Disaster Recovery 101: Architecting Auto-Failovers for MySQL and WooCommerce Deployments on DigitalOcean

Establishing a High-Availability MySQL Cluster with Orchestrator

For mission-critical applications like WooCommerce, a single MySQL instance is a single point of failure. Architecting for high availability (HA) necessitates a robust failover strategy. We’ll leverage Orchestrator, a popular MySQL replication topology manager, to automate this process. Orchestrator monitors replication health and can automatically promote a replica to a primary in case of failure.

Our setup will involve at least two MySQL nodes (a primary and a replica) and a separate Orchestrator instance. For simplicity, we’ll assume a basic topology. In a production environment, you’d want multiple Orchestrator instances for their own HA and potentially more MySQL nodes for read scaling and redundancy.

Orchestrator Installation and Configuration

Orchestrator can be installed from source or via pre-built packages. For this example, we’ll assume a Debian/Ubuntu-based system. The configuration file, typically /etc/orchestrator/orchestrator.conf.json, is crucial.

Key configuration parameters include:

  • MySQLTopologyUser and MySQLTopologyPassword: Credentials for Orchestrator to connect to MySQL instances. These users need sufficient privileges (e.g., REPLICATION SLAVE, REPLICATION CLIENT, SELECT).
  • DiscoveryPollSeconds: How often Orchestrator scans the topology.
  • FailureDetectionPeriodSeconds: How long a failure must persist before triggering an action.
  • PromotionRule: Defines the logic for promoting a replica (e.g., ‘prefer-most-recent-master-check’).
  • ListenAddress: The IP address Orchestrator listens on.
  • HTTPPort: The port for the Orchestrator web UI and API.
  • MySQLOrchestratorHostPort: The host and port of the MySQL instance Orchestrator itself connects to for topology discovery (often a dedicated user or the primary).

Here’s a sample orchestrator.conf.json:

Ensure the orchestrator user has the necessary MySQL privileges. You can create this user and grant permissions like so:

-- On each MySQL node:
CREATE USER 'orchestrator'@'%' IDENTIFIED BY 'your_secure_password';
GRANT REPLICATION SLAVE, REPLICATION CLIENT, SELECT ON *.* TO 'orchestrator'@'%';
FLUSH PRIVILEGES;
{
  "Debug": true,
  "ListenAddress": "0.0.0.0",
  "HTTPPort": 8080,
  "MySQLTopologyUser": "orchestrator",
  "MySQLTopologyPassword": "your_secure_password",
  "MySQLOrchestratorHostPort": "127.0.0.1:3306",
  "DiscoveryPollSeconds": 10,
  "FailureDetectionPeriodSeconds": 30,
  "PromotionRule": "prefer-most-recent-master-check",
  "RecoveryPeriodBlockSeconds": 3600,
  "DetectClusterAlias": true,
  "SlaveLagQuery": "SELECT * FROM mysql.slave_replica_status WHERE ServerId = ?",
  "SlaveLagQueryIntervalSeconds": 5,
  "GlobalWriteRecoveryPeriodBlockSeconds": 3600,
  "AutoOrchestrateOnFailure": true,
  "PostUnsuccessfulFailoverProcesses": [
    "discover"
  ],
  "PostSuccessfulFailoverProcesses": [
    "discover"
  ],
  "PrePromotionProcesses": [],
  "PostPromotionProcesses": [
    "discover"
  ],
  "PreFailoverProcesses": [],
  "PostFailoverProcesses": [
    "discover"
  ],
  "PostRelocationProcesses": [
    "discover"
  ],
  "PostUnsuccessfulRecoveryProcesses": [
    "discover"
  ],
  "PostSuccessfulRecoveryProcesses": [
    "discover"
  ],
  "PostRelocationProcesses": [
    "discover"
  ],
  "MySQLBinlogPlayerHostPort": "127.0.0.1:3306",
  "MySQLBinlogPlayerUser": "orchestrator",
  "MySQLBinlogPlayerPassword": "your_secure_password",
  "MySQLBinlogPlayerPort": 3306,
  "MySQLBinlogPlayerDatabase": "mysql",
  "MySQLBinlogPlayerTable": "binlog_player",
  "MySQLBinlogPlayerSlaveIdOffset": 1000000,
  "MySQLBinlogPlayerSlaveIdIncrement": 1,
  "MySQLBinlogPlayerSlaveIdMax": 2000000,
  "MySQLBinlogPlayerSlaveIdPoolSize": 100,
  "MySQLBinlogPlayerSlaveIdPoolRefreshSeconds": 300,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxAttempts": 5,
  "MySQLBinlogPlayerSlaveIdPoolRefreshRetryIntervalSeconds": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshTimeoutSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxTimeoutSeconds": 300,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds": 60,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryAttempts": 10,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryTimeoutSeconds": 600,
  "MySQLBinlogPlayerSlaveIdPoolRefreshMaxRetryIntervalSeconds":