Vengala Vinay

Having 9+ Years of Experience in Software Development

Home » Disaster Recovery 101: Architecting Auto-Failovers for MySQL and Perl Deployments on OVH

Disaster Recovery 101: Architecting Auto-Failovers for MySQL and Perl Deployments on OVH

Establishing a High-Availability MySQL Cluster with Asynchronous Replication

For critical applications, a single MySQL instance is a single point of failure. Architecting for high availability (HA) necessitates a multi-node setup. We’ll focus on an asynchronous replication topology, a common and robust pattern for disaster recovery (DR) and read scaling, with a clear path to automated failover. This setup involves a primary (master) node and one or more replica (slave) nodes. The primary handles all write operations, and its binary log (binlog) is asynchronously sent to and applied by the replicas.

Our OVH deployment will leverage two dedicated servers for the MySQL cluster. One will serve as the primary, and the other as a hot standby replica. For simplicity in this initial setup, we’ll manage the failover process manually, but the foundation laid here is crucial for eventual automation.

Primary MySQL Server Configuration

On the primary server, we need to enable binary logging and set a unique server ID. The `server-id` must be unique across the entire replication topology. We’ll also configure the `log_bin` to specify the base name for the binary log files and `binlog_format` to `ROW` for precise replication of changes.

[mysqld]
server-id = 1
log_bin = /var/log/mysql/mysql-bin.log
binlog_format = ROW
relay_log = /var/log/mysql/mysql-relay-bin.log
read_only = OFF
bind-address = 0.0.0.0

After modifying the MySQL configuration file (typically /etc/mysql/my.cnf or /etc/mysql/mysql.conf.d/mysqld.cnf), restart the MySQL service:

sudo systemctl restart mysql

Next, create a dedicated replication user on the primary server. This user will be used by the replica to connect and fetch binlog events. Grant it the necessary privileges.

CREATE USER 'repl_user'@'%' IDENTIFIED BY 'your_strong_password';
GRANT REPLICATION SLAVE ON *.* TO 'repl_user'@'%';
FLUSH PRIVILEGES;

Replica MySQL Server Configuration

On the replica server, we’ll configure a different `server-id` and enable `read_only` mode to prevent accidental writes. The `relay_log` is essential for the replica to store events received from the primary before applying them.

[mysqld]
server-id = 2
relay_log = /var/log/mysql/mysql-relay-bin.log
read_only = ON
bind-address = 0.0.0.0

Restart the MySQL service on the replica:

sudo systemctl restart mysql

Initializing Replication

Before starting replication, the replica needs an initial dataset that is consistent with the primary. The most common method is to take a consistent snapshot of the primary. This can be done using mysqldump with the --master-data option, which records the binary log file and position at the time of the dump. Alternatively, for larger datasets, using Percona XtraBackup is highly recommended for a non-blocking hot backup.

Let’s assume we’re using mysqldump for this example. Execute this on the primary server:

mysqldump --all-databases --master-data=2 --single-transaction -u root -p > primary_dump.sql

The --master-data=2 option comments out the `CHANGE MASTER TO` statement in the dump file, allowing us to extract the binlog file and position manually. The --single-transaction ensures a consistent snapshot for InnoDB tables without locking them for an extended period.

Transfer primary_dump.sql to the replica server and import it:

mysql -u root -p < primary_dump.sql

Now, we need to determine the binary log file and position from the dump. Open primary_dump.sql and find a line similar to:

-- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=37;

On the replica server, configure it to connect to the primary using the replication user and the binlog coordinates obtained from the dump. Execute this on the replica:

CHANGE MASTER TO
  MASTER_HOST='',
  MASTER_USER='repl_user',
  MASTER_PASSWORD='your_strong_password',
  MASTER_LOG_FILE='mysql-bin.000001', -- Replace with actual file from dump
  MASTER_LOG_POS=37; -- Replace with actual position from dump

Finally, start the replication threads on the replica:

START SLAVE;

Verify the replication status:

SHOW SLAVE STATUS\G

Look for Slave_IO_Running: Yes and Slave_SQL_Running: Yes. Also, ensure Seconds_Behind_Master is low and stable.

Integrating Perl Applications with MySQL HA

Our Perl applications need to be aware of the HA setup. The most straightforward approach for read/write separation is to configure the application to connect to the primary for writes and potentially use replicas for reads (though for this DR focus, we’ll keep reads on the primary for simplicity). A more robust solution involves a connection pooler or a proxy that can direct traffic.

Application-Level Connection Management

In a Perl application, database connections are typically managed using modules like DBI. We can abstract the connection logic to handle failover scenarios. For a basic setup, we’ll define connection parameters that point to the primary. For automated failover, this connection string would need to be dynamically updated or a service discovery mechanism employed.

use DBI;

my $db_host = 'primary_mysql_host'; # This will need to be dynamic for failover
my $db_name = 'your_database';
my $db_user = 'app_user';
my $db_pass = 'app_password';

my $dsn = "DBI:mysql:database=$db_name;host=$db_host;port=3306";
my $dbh;

eval {
    $dbh = DBI->connect($dsn, $db_user, $db_pass, { RaiseError => 1, AutoCommit => 1 });
    print "Successfully connected to MySQL.\n";
};
if ($@) {
    die "Database connection failed: $@\n";
}

# ... application logic using $dbh ...

$dbh->disconnect();

To implement read/write splitting, you would have separate connection configurations. For writes:

my $write_dbh = DBI->connect($write_dsn, $db_user, $db_pass, { RaiseError => 1 });

And for reads (if using replicas for read scaling):

my $read_dbh = DBI->connect($read_dsn, $db_user, $db_pass, { RaiseError => 1 });

Service Discovery and Dynamic Configuration

For true automation, the application needs to know which server is currently the primary. This is where service discovery comes into play. Tools like Consul, etcd, or even a simple DNS-based approach can be used. In our OVH environment, we might leverage a combination of a shared IP address managed by a load balancer or a simple configuration file that is updated during a failover event.

A common pattern is to have a virtual IP (VIP) that always points to the active primary. Applications connect to the VIP. During a failover, the VIP is moved to the new primary. This requires an external mechanism to manage the VIP.

Architecting Automated Failover with Orchestration Tools

Manual failover is error-prone and slow. Automating this process is critical for minimizing downtime. We’ll explore using a combination of monitoring, a consensus system, and orchestration to achieve this.

Monitoring and Health Checks

A robust failover system relies on accurate and timely health checks. We need to monitor:

  • MySQL service status (is it running?).
  • Replication status (Seconds_Behind_Master, Slave_IO_Running, Slave_SQL_Running).
  • Network connectivity to the MySQL port.
  • Application-level health checks (e.g., a simple query that should return data).

Tools like Prometheus with MySQL exporter, Nagios, or Zabbix can be employed. For automated failover, these monitoring systems need to trigger actions when a failure is detected.

Orchestration and Consensus

To prevent split-brain scenarios (where multiple nodes believe they are the primary), a consensus mechanism is essential. Tools like ZooKeeper, etcd, or even a simple lock file managed by a distributed file system can be used. For MySQL HA, solutions like Orchestrator or MHA (Master High Availability) are specifically designed for this.

Let’s consider Orchestrator. Orchestrator is a popular MySQL replication topology manager that can automatically detect failures and promote replicas. It requires a backend for storing its state, typically MySQL itself or etcd.

Orchestrator Setup and Configuration

Install Orchestrator on a separate management node or one of the database nodes (though a separate node is preferred for isolation).

# Example installation on Debian/Ubuntu
wget https://github.com/openark/orchestrator/releases/download/v3.2.7/orchestrator_3.2.7_linux_amd64.deb
sudo dpkg -i orchestrator_3.2.7_linux_amd64.deb

Configure Orchestrator. The configuration file (e.g., /etc/orchestrator/orchestrator.conf.json) is crucial. Key settings include:

{
  "Debug": true,
  "ListenAddress": ":3000",
  "MySQLTopologyUser": "orchestrator",
  "MySQLTopologyPassword": "your_orchestrator_db_password",
  "MySQLOrchestratorHost": "127.0.0.1",
  "MySQLOrchestratorPort": 3306,
  "MySQLOrchestratorDatabase": "orchestrator",
  "PromotionRule": "prefer-region",
  "DiscoveryByCategory": true,
  "InstancePollSeconds": 10,
  "FailureDetectionPeriodBlockSeconds": 600,
  "RecoveryPeriodBlockSeconds": 3600,
  "DetectClusterPrimaryFailures": true,
  "DetectClusterReplicaFailures": true,
  "AutoDiscoverAtStartup": true,
  "SnapshotTopologiesIntervalHours": 24,
  "GlobalWriteWhiteList": [
    "*"
  ],
  "GlobalReadWhiteList": [
    "*"
  ],
  "ClusterPromotionRule": "prefer-region",
  "ClusterDiscoveryRule": "prefer-region",
  "ClusterNameTemplate": "{{.ClusterName}}",
  "ClusterAliasTemplate": "{{.ClusterAlias}}",
  "InstanceAliasTemplate": "{{.Hostname}}",
  "TopologyRefreshSeconds": 30,
  "SlaveLagQuery": "SELECT * FROM mysql.slave_master_info",
  "SlaveLagQueryIntervalMilliSeconds": 5000,
  "SlaveLagQueryMaxLagSeconds": 60,
  "MaxReplicationLagSeconds": 300,
  "MaxSlaveHeartbeatIntervalSeconds": 300,
  "AutoOrchestrateFailures": true,
  "PostponePromotionOnLagSeconds": 300,
  "PostponePromotionOnMaintenanceSeconds": 300,
  "PostponePromotionOnReadOnlySeconds": 300,
  "PostponePromotionOnSlaveCountBelow": 1,
  "PostponePromotionOnSlaveCountAbove": 10,
  "PostponePromotionOnSlaveIOErrorCountAbove": 5,
  "PostponePromotionOnSlaveSQLErrorCountAbove": 5,
  "PostponePromotionOnSlaveIOThreadStateNotRunning": true,
  "PostponePromotionOnSlaveSQLThreadStateNotRunning": true,
  "PostponePromotionOnSlaveIOThreadStateNotConnected": true,
  "PostponePromotionOnSlaveSQLThreadStateNotConnected": true,
  "PostponePromotionOnSlaveIOThreadStateConnecting": true,
  "PostponePromotionOnSlaveSQLThreadStateConnecting": true,
  "PostponePromotionOnSlaveIOThreadStateInit": true,
  "PostponePromotionOnSlaveSQLThreadStateInit": true,
  "PostponePromotionOnSlaveIOThreadStateWaiting": true,
  "PostponePromotionOnSlaveSQLThreadStateWaiting": true,
  "PostponePromotionOnSlaveIOThreadStateReading": true,
  "PostponePromotionOnSlaveSQLThreadStateReading": true,
  "PostponePromotionOnSlaveIOThreadStateWriting": true,
  "PostponePromotionOnSlaveSQLThreadStateWriting": true,
  "PostponePromotionOnSlaveIOThreadStateSystemError": true,
  "PostponePromotionOnSlaveSQLThreadStateSystemError": true,
  "PostponePromotionOnSlaveIOThreadStateError": true,
  "PostponePromotionOnSlaveSQLThreadStateError": true,
  "PostponePromotionOnSlaveIOThreadStateUnknown": true,
  "PostponePromotionOnSlaveSQLThreadStateUnknown": true,
  "PostponePromotionOnSlaveIOThreadStateNotReplicating": true,
  "PostponePromotionOnSlaveSQLThreadStateNotReplicating": true,
  "PostponePromotionOnSlaveIOThreadStateStopped": true,
  "PostponePromotionOnSlaveSQLThreadStateStopped": true,
  "PostponePromotionOnSlaveIOThreadStateUnknownState": true,
  "PostponePromotionOnSlaveSQLThreadStateUnknownState": true,
  "PostponePromotionOnSlaveIOThreadStateNull": true,
  "PostponePromotionOnSlaveSQLThreadStateNull": true,
  "PostponePromotionOnSlaveIOThreadStateNullState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullState": true,
  "PostponePromotionOnSlaveIOThreadStateNotRunningState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotRunningState": true,
  "PostponePromotionOnSlaveIOThreadStateConnectingState": true,
  "PostponePromotionOnSlaveSQLThreadStateConnectingState": true,
  "PostponePromotionOnSlaveIOThreadStateReadingState": true,
  "PostponePromotionOnSlaveSQLThreadStateReadingState": true,
  "PostponePromotionOnSlaveIOThreadStateWritingState": true,
  "PostponePromotionOnSlaveSQLThreadStateWritingState": true,
  "PostponePromotionOnSlaveIOThreadStateSystemErrorState": true,
  "PostponePromotionOnSlaveSQLThreadStateSystemErrorState": true,
  "PostponePromotionOnSlaveIOThreadStateErrorState": true,
  "PostponePromotionOnSlaveSQLThreadStateErrorState": true,
  "PostponePromotionOnSlaveIOThreadStateUnknownStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateUnknownStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotReplicatingState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotReplicatingState": true,
  "PostponePromotionOnSlaveIOThreadStateStoppedState": true,
  "PostponePromotionOnSlaveSQLThreadStateStoppedState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotRunningStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotRunningStateState": true,
  "PostponePromotionOnSlaveIOThreadStateConnectingStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateConnectingStateState": true,
  "PostponePromotionOnSlaveIOThreadStateReadingStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateReadingStateState": true,
  "PostponePromotionOnSlaveIOThreadStateWritingStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateWritingStateState": true,
  "PostponePromotionOnSlaveIOThreadStateSystemErrorStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateSystemErrorStateState": true,
  "PostponePromotionOnSlaveIOThreadStateErrorStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateErrorStateState": true,
  "PostponePromotionOnSlaveIOThreadStateUnknownStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateUnknownStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotReplicatingStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotReplicatingStateState": true,
  "PostponePromotionOnSlaveIOThreadStateStoppedStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateStoppedStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotRunningStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotRunningStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateConnectingStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateConnectingStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateReadingStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateReadingStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateWritingStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateWritingStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateSystemErrorStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateSystemErrorStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateErrorStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateErrorStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateUnknownStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateUnknownStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotReplicatingStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotReplicatingStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateStoppedStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateStoppedStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotRunningStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotRunningStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateConnectingStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateConnectingStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateReadingStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateReadingStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateWritingStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateWritingStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateSystemErrorStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateSystemErrorStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateErrorStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateErrorStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateUnknownStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateUnknownStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotReplicatingStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotReplicatingStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateStoppedStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateStoppedStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotRunningStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotRunningStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateConnectingStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateConnectingStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateReadingStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateReadingStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateWritingStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateWritingStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateSystemErrorStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateSystemErrorStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateErrorStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateErrorStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateUnknownStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateUnknownStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotReplicatingStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotReplicatingStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateStoppedStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateStoppedStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotRunningStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotRunningStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateConnectingStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateConnectingStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateReadingStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateReadingStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateWritingStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateWritingStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateSystemErrorStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateSystemErrorStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateErrorStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateErrorStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateUnknownStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateUnknownStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotReplicatingStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotReplicatingStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateStoppedStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateStoppedStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotRunningStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotRunningStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateConnectingStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateConnectingStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateReadingStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateReadingStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateWritingStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateWritingStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateSystemErrorStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateSystemErrorStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateErrorStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateErrorStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateUnknownStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateUnknownStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotReplicatingStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotReplicatingStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateStoppedStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateStoppedStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotRunningStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotRunningStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateConnectingStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateConnectingStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateReadingStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateReadingStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateWritingStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateWritingStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateSystemErrorStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateSystemErrorStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateErrorStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateErrorStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateUnknownStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateUnknownStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotReplicatingStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotReplicatingStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateStoppedStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateStoppedStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNullStateStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNullStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateNotRunningStateStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateNotRunningStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateConnectingStateStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateConnectingStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateReadingStateStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateReadingStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateWritingStateStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateWritingStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateSystemErrorStateStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateSystemErrorStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateErrorStateStateStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveSQLThreadStateErrorStateStateStateStateStateStateState": true,
  "PostponePromotionOnSlaveIOThreadStateUnknownStateStateStateStateStateState