Disaster Recovery 101: Architecting Auto-Failovers for Redis and C Deployments on Google Cloud
Architecting Redis Sentinel for High Availability on Google Cloud
Achieving robust disaster recovery for stateful services like Redis necessitates an automated failover strategy. On Google Cloud Platform (GCP), this translates to leveraging managed services where possible or meticulously configuring self-managed solutions. For Redis, the de facto standard for high availability is Redis Sentinel. This section details the architecture and configuration of Redis Sentinel within a GCP environment, focusing on automatic failover.
Our architecture will consist of a primary Redis instance, one or more replica instances, and a quorum of Redis Sentinel processes. These Sentinels monitor the primary and replicas, orchestrate failover when the primary becomes unavailable, and reconfigure clients to point to the new primary. For optimal resilience, Sentinels should be deployed across different availability zones within a GCP region.
Setting up Redis Instances
We’ll assume you have basic Redis instances running on Compute Engine VMs. For simplicity, we’ll use a single primary and a single replica, but a production setup would involve at least two replicas for read scaling and redundancy. Ensure your Redis instances are configured for replication.
On your primary Redis VM (e.g., `redis-primary-vm`):
# redis.conf port 6379 daemonize yes pidfile /var/run/redis_6379.pid logfile /var/log/redis/redis-server.log dir /var/lib/redis # For Sentinel to connect, bind to a public IP or a specific internal IP # that Sentinels can reach. If using GCP firewall rules, ensure this is allowed. bind 0.0.0.0 protected-mode no # Persistence settings (e.g., RDB or AOF) are crucial for data durability # but not directly for Sentinel's failover mechanism itself. # save 900 1 # save 300 10 # save 60 10000 # appendonly yes
On your replica Redis VM (e.g., `redis-replica-vm`):
# redis.conf port 6379 daemonize yes pidfile /var/run/redis_6379.pid logfile /var/log/redis/redis-server.log dir /var/lib/redis bind 0.0.0.0 protected-mode no # Replication settings replicaof <primary-private-ip> 6379 # replica-read-only yes # Recommended for replicas not serving direct client reads
Replace <primary-private-ip> with the internal IP address of your primary Redis VM. Ensure GCP firewall rules allow traffic on port 6379 between these VMs.
Configuring Redis Sentinel
Redis Sentinel configuration is stored in a separate file, typically `sentinel.conf`. We need at least three Sentinel instances for a quorum. Deploy these on separate VMs, ideally in different GCP availability zones.
On each Sentinel VM (e.g., `redis-sentinel-vm-1`, `redis-sentinel-vm-2`, `redis-sentinel-vm-3`):
# sentinel.conf port 26379 daemonize yes pidfile /var/run/redis-sentinel.pid logfile /var/log/redis/sentinel.log dir /var/lib/redis # Monitor the primary Redis master. # 'mymaster' is the name of the master Redis instance. # 10 is the quorum: the number of Sentinels that must agree that the master is down. # 2000 is the failover timeout: how long Redis Sentinel waits before starting to failover. sentinel monitor mymaster <primary-private-ip> 6379 2 # This is the number of replicas that should be promoted to master. # In a typical setup, this is 1. sentinel parallel-sync-slaves mymaster 1 # The time in milliseconds that the master must be unreachable for it to be # considered by Sentinel as down. sentinel down-after-milliseconds mymaster 5000 # The time in seconds that Sentinel will wait before starting the failover # procedure. This is the time after the master is marked as Subjectively Down # (SDOWN) by a quorum of Sentinels. sentinel failover-timeout mymaster 10000 # Optional: Configure Sentinel to notify an external script on failover events. # sentinel notification-script mymaster /path/to/your/notification-script.sh # Optional: Configure Sentinel to run a script to reconfigure clients. # sentinel client-reconfig-script mymaster /path/to/your/client-reconfig-script.sh # Bind Sentinel to an IP address that clients and other Sentinels can reach. # If Sentinels need to communicate with each other across VMs, this is crucial. bind 0.0.0.0
Key parameters:
sentinel monitor mymaster <primary-private-ip> 6379 2: This line tells Sentinel to monitor a master named ‘mymaster’ at the specified IP and port. The ‘2’ is the quorum – at least two Sentinels must agree the master is down for a failover to be initiated. For a 3-Sentinel setup, a quorum of 2 is typical.sentinel down-after-milliseconds mymaster 5000: If a Sentinel doesn’t receive a PING reply from the master within 5000ms, it marks the master as Subjectively Down (SDOWN).sentinel failover-timeout mymaster 10000: After the master is marked as SDOWN by a quorum, this is the time Sentinel waits before initiating the failover process.sentinel parallel-sync-slaves mymaster 1: During failover, this limits the number of replicas that can be reconfigured to sync with the new master simultaneously.
Ensure GCP firewall rules allow traffic on port 26379 between your Sentinel VMs and on port 6379 between Sentinels and your Redis instances.
Starting and Verifying Sentinel
Start the Redis Sentinel processes:
redis-sentinel /etc/redis/sentinel.conf
You can verify the Sentinel status and configuration using `redis-cli` connected to a Sentinel instance:
redis-cli -p 26379
127.0.0.1:26379> SENTINEL masters
1) 1) "name"
2) "mymaster"
3) "ip"
4) "<primary-private-ip>"
5) "port"
6) "6379"
7) "runid"
8) "..."
9) "flags"
10) "master"
11) "num-slaves"
12) "1"
13) "num-other-sentinels"
14) "2"
15) "quorum"
16) "2"
17) "config-epoch"
18) "0"
19) "role-reported-time"
20) "..."
21) "master-host"
22) "<primary-private-ip>"
23) "master-port"
24) "6379"
25) "down-after-milliseconds"
26) "5000"
27) "failover-timeout"
28) "10000"
29) "parallel-sync-slaves"
30) "1"
127.0.0.1:26379> SENTINEL slaves mymaster
1) 1) "name"
2) "slave-0"
3) "ip"
4) "<replica-private-ip>"
5) "port"
6) "6379"
7) "runid"
8) "..."
9) "flags"
10) "slave"
11) "master-link-down-since-seconds"
12) "0"
13) "master-link-status"
14) "up"
15) "replication-offset"
16) "..."
17) "master-host"
18) "<primary-private-ip>"
19) "master-port"
20) "6379"
21) "slave-priority"
22) "100"
23) "ssl-port"
24) "6379"
To test failover, stop the primary Redis instance:
# On the primary Redis VM sudo systemctl stop redis-server # or redis-cli shutdown
Monitor the Sentinel logs and use `redis-cli -p 26379 SENTINEL masters` to observe the failover process. The replica should be promoted to master, and the Sentinels will update their configuration.
Automating Client Reconfiguration with C and GCP Metadata
When a Redis failover occurs, your application clients need to be directed to the new master. Hardcoding IP addresses is brittle. A robust solution involves dynamically discovering the current master’s IP. We can achieve this by having Sentinels update a central, discoverable source of truth, or by having clients query Sentinels directly. For this example, we’ll explore a pattern where a C application, running on GCP Compute Engine, queries Sentinel to find the current master and updates its connection pool.
Client-Side Sentinel Querying in C
The Redis Sentinel API provides commands to query the current master for a given monitored master name. The `SENTINEL get-master-addr-by-name <master-name>` command is key here.
We’ll use the `hiredis` library for C to interact with Redis and Sentinel. First, ensure you have `hiredis` installed (e.g., `sudo apt-get install libhiredis-dev`).
Consider a C program that periodically checks the Redis master and updates a connection pool. For simplicity, this example shows a single check. In a real application, this would be part of a background thread or a connection manager.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <hiredis/hiredis.h>
// Function to get the current master IP address from Sentinel
char* get_redis_master_ip(const char* sentinel_host, int sentinel_port, const char* master_name) {
redisContext *c = NULL;
redisReply *reply = NULL;
char *master_ip = NULL;
// Connect to Sentinel
c = redisConnect(sentinel_host, sentinel_port);
if (c == NULL || c->err) {
if (c) {
fprintf(stderr, "Sentinel connection error: %s\n", c->errstr);
redisFree(c);
} else {
fprintf(stderr, "Sentinel connection error: can't allocate redis context\n");
}
return NULL;
}
// Send the SENTINEL get-master-addr-by-name command
reply = redisCommand(c, "SENTINEL get-master-addr-by-name %s", master_name);
if (reply == NULL) {
fprintf(stderr, "Redis command error: %s\n", c->errstr);
redisFree(c);
return NULL;
}
// Parse the reply. Expected format: [ "ip_address", "port_number" ]
if (reply->type == REDIS_REPLY_ARRAY && reply->elements == 2) {
if (reply->element[0]->type == REDIS_REPLY_STRING) {
master_ip = strdup(reply->element[0]->str); // Duplicate the IP string
}
} else {
fprintf(stderr, "Unexpected reply from Sentinel: type=%d, elements=%zu\n", reply->type, reply->elements);
}
freeReplyObject(reply);
redisFree(c);
return master_ip;
}
// Placeholder for connection pool management
void update_redis_connection_pool(const char* new_master_ip) {
printf("Updating connection pool to point to new master: %s\n", new_master_ip);
// In a real application, you would:
// 1. Close existing connections to the old master.
// 2. Establish new connections to the new_master_ip.
// 3. Update internal pointers or data structures for the connection pool.
}
int main() {
// These should ideally be configurable or fetched from GCP metadata/environment variables
const char* sentinel_host = "redis-sentinel-vm-1-ip"; // IP of one of your Sentinel VMs
const int sentinel_port = 26379;
const char* redis_master_name = "mymaster"; // The name defined in sentinel.conf
char* current_master_ip = get_redis_master_ip(sentinel_host, sentinel_port, redis_master_name);
if (current_master_ip) {
printf("Current Redis master IP: %s\n", current_master_ip);
// In a real application, you'd compare this to your currently known master
// and trigger an update if it has changed.
update_redis_connection_pool(current_master_ip);
free(current_master_ip); // Free the duplicated string
} else {
fprintf(stderr, "Failed to retrieve Redis master IP.\n");
return 1;
}
return 0;
}
To compile this:
gcc your_program.c -o redis_client -lhiredis
Leveraging GCP Metadata for Sentinel Host Discovery
Instead of hardcoding Sentinel IP addresses, we can use GCP’s instance metadata service. This allows your application to discover the IP addresses of Sentinel instances without manual configuration. A common pattern is to tag your Sentinel VMs with a specific label (e.g., `redis-role: sentinel`) and then use the GCP API to list instances with that tag and retrieve their internal IP addresses.
However, directly querying the GCP API from a C application can add complexity (HTTP client, JSON parsing). A simpler approach for this C example is to pre-populate a configuration file or environment variables with the IPs of *all* Sentinel instances. The C code would then iterate through these IPs, attempting to connect to the first available Sentinel. This provides resilience if one Sentinel VM becomes unreachable.
Let’s modify the C code to accept a list of Sentinel hosts and try them sequentially.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <hiredis/hiredis.h>
// Function to get the current master IP address from Sentinel
char* get_redis_master_ip(const char* sentinel_host, int sentinel_port, const char* master_name) {
redisContext *c = NULL;
redisReply *reply = NULL;
char *master_ip = NULL;
c = redisConnect(sentinel_host, sentinel_port);
if (c == NULL || c->err) {
if (c) {
fprintf(stderr, "Sentinel connection error to %s:%d: %s\n", sentinel_host, sentinel_port, c->errstr);
redisFree(c);
} else {
fprintf(stderr, "Sentinel connection error to %s:%d: can't allocate redis context\n", sentinel_host, sentinel_port);
}
return NULL;
}
reply = redisCommand(c, "SENTINEL get-master-addr-by-name %s", master_name);
if (reply == NULL) {
fprintf(stderr, "Redis command error to %s:%d: %s\n", sentinel_host, sentinel_port, c->errstr);
redisFree(c);
return NULL;
}
if (reply->type == REDIS_REPLY_ARRAY && reply->elements == 2) {
if (reply->element[0]->type == REDIS_REPLY_STRING) {
master_ip = strdup(reply->element[0]->str);
}
} else {
fprintf(stderr, "Unexpected reply from Sentinel %s:%d: type=%d, elements=%zu\n", sentinel_host, sentinel_port, reply->type, reply->elements);
}
freeReplyObject(reply);
redisFree(c);
return master_ip;
}
// Function to try multiple Sentinels until one responds successfully
char* find_redis_master_ip(const char* sentinel_hosts[], int num_sentinels, int sentinel_port, const char* master_name) {
for (int i = 0; i < num_sentinels; ++i) {
char* ip = get_redis_master_ip(sentinel_hosts[i], sentinel_port, master_name);
if (ip) {
printf("Successfully queried Sentinel at %s\n", sentinel_hosts[i]);
return ip; // Return the first successful result
}
}
return NULL; // Failed to connect to any Sentinel
}
void update_redis_connection_pool(const char* new_master_ip) {
printf("Updating connection pool to point to new master: %s\n", new_master_ip);
// ... connection pool logic ...
}
int main() {
// In a real GCP deployment, these IPs would be populated from:
// 1. GCP Metadata API (via a helper script or library)
// 2. Kubernetes service discovery
// 3. A configuration file managed by deployment tools (Terraform, Ansible)
const char* sentinel_ips[] = {
"10.128.0.10", // Internal IP of Sentinel VM 1 (e.g., in zone a)
"10.128.1.10", // Internal IP of Sentinel VM 2 (e.g., in zone b)
"10.128.2.10" // Internal IP of Sentinel VM 3 (e.g., in zone c)
};
int num_sentinels = sizeof(sentinel_ips) / sizeof(sentinel_ips[0]);
const int sentinel_port = 26379;
const char* redis_master_name = "mymaster";
char* current_master_ip = find_redis_master_ip(sentinel_ips, num_sentinels, sentinel_port, redis_master_name);
if (current_master_ip) {
printf("Current Redis master IP: %s\n", current_master_ip);
update_redis_connection_pool(current_master_ip);
free(current_master_ip);
} else {
fprintf(stderr, "Failed to retrieve Redis master IP from any Sentinel.\n");
return 1;
}
return 0;
}
This approach makes the client more resilient to individual Sentinel failures. The `update_redis_connection_pool` function is where the core logic for managing your application’s Redis connections resides. This could involve a thread that periodically calls `find_redis_master_ip` and updates the pool if the master IP changes.
Automating Failover for C Deployments on GCP
For C applications deployed on GCP, achieving automated failover involves integrating the application’s lifecycle management with GCP’s infrastructure. This typically means using GCP’s managed instance groups (MIGs) and health checks.
Managed Instance Groups and Health Checks
A Managed Instance Group (MIG) allows you to manage a group of identical VM instances. MIGs can automatically recreate instances that fail, ensuring your application remains available. To integrate this with Redis failover, we need a health check that accurately reflects the application’s ability to reach and use Redis.
1. Application Health Check Endpoint:
Your C application should expose an HTTP health check endpoint. This endpoint would perform a quick Redis operation (e.g., `PING` or `GET` on a known key) and return an HTTP 200 OK if successful, or an HTTP 5xx error if it cannot reach Redis or perform the operation.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <hiredis/hiredis.h>
#include <microhttpd.h> // For HTTP server
// ... (get_redis_master_ip and find_redis_master_ip functions from above) ...
// Global Redis context for health check
redisContext *health_check_ctx = NULL;
char *current_redis_master_ip = NULL; // Track the current master IP
// Function to initialize/update Redis connection for health checks
void initialize_redis_health_check_connection(const char* master_ip) {
if (current_redis_master_ip && strcmp(current_redis_master_ip, master_ip) == 0) {
// IP hasn't changed, no need to re-initialize if context is valid
if (health_check_ctx && !health_check_ctx->err) return;
}
printf("Initializing/Updating health check Redis connection to: %s\n", master_ip);
if (health_check_ctx) {
redisFree(health_check_ctx);
health_check_ctx = NULL;
}
current_redis_master_ip = strdup(master_ip); // Update tracked IP
health_check_ctx = redisConnect(master_ip, 6379);
if (health_check_ctx == NULL || health_check_ctx->err) {
fprintf(stderr, "Health check Redis connection error to %s: %s\n", master_ip, health_check_ctx ? health_check_ctx->errstr : "can't allocate context");
if (health_check_ctx) {
redisFree(health_check_ctx);
health_check_ctx = NULL;
}
} else {
printf("Health check Redis connection established to %s.\n", master_ip);
}
}
// HTTP request handler
static int ahc_request_handler(void *cls, struct MHD_Connection *connection,
const char *url, const char *method,
const char *version, const char *upload_data,
size_t *upload, unsigned int *flags) {
if (strcmp(method, "GET") != 0) {
return MHD_NO; // Only accept GET requests
}
// Perform a PING to Redis
if (health_check_ctx && !health_check_ctx->err) {
redisReply *reply = redisCommand(health_check_ctx, "PING");
if (reply && strcmp(reply->str, "PONG") == 0) {
freeReplyObject(reply);
// Redis is reachable and responding
const char *response = "OK";
struct MHD_Response *mhd_response =
MHD_create_response_from_buffer(strlen(response), (void *)response, MHD_RESPMEM_PERSISTENT);
MHD_add_header(mhd_response, MHD_HTTP_HEADER_CONTENT_TYPE, "text/plain");
int ret = MHD_queue_response(connection, MHD_HTTP_OK, mhd_response);
MHD_destroy_response(mhd_response);
return ret;
} else {
fprintf(stderr, "Redis PING failed or returned unexpected response.\n");
if (reply) freeReplyObject(reply);
// Connection might be stale, try to re-establish
initialize_redis_health_check_connection(current_redis_master_ip); // Attempt re-init
}
}
// If no valid connection or PING failed
const char *response = "Service Unavailable";
struct MHD_Response *mhd_response =
MHD_create_response_from_buffer(strlen(response), (void *)response, MHD_RESPMEM_PERSISTENT);
MHD_add_header(mhd_response, MHD_HTTP_HEADER_CONTENT_TYPE, "text/plain");
int ret = MHD_queue_response(connection, MHD_HTTP_SERVICE_UNAVAILABLE, mhd_response);
MHD_destroy_response(mhd_response);
return ret;
}
// Background thread to periodically check Sentinel and update connection
void *redis_monitor_thread(void *arg) {
const char* sentinel_ips[] = { /* ... your sentinel IPs ... */ };
int num_sentinels = sizeof(sentinel_ips) / sizeof(sentinel_ips[0]);
const int sentinel_port = 26379;
const char* redis_master_name = "mymaster";
while (1) {
char* master_ip = find_redis_master_ip(sentinel_ips, num_sentinels, sentinel_port, redis_master_name);
if (master_ip) {
initialize_redis_health_check_connection(master_ip);
free(master_ip);
} else {
fprintf(stderr, "Failed to find Redis master IP in monitor thread.\n");
// Ensure health check fails if master cannot be found
if (health_check_ctx) {
redisFree(health_check_ctx);
health_check_ctx = NULL;
}
}
sleep(10); // Check every 10 seconds
}
return NULL;
}
int main() {
// Start the monitoring thread
pthread_t monitor_tid;
if (pthread_create(&monitor_tid, NULL, redis_monitor_thread, NULL) != 0) {
perror("Failed to create Redis monitor thread");
return 1;
}
// Start the HTTP server for health checks
struct MHD_Daemon *daemon;
const int http_port = 8080; // Port for health checks
daemon = MHD_start_daemon(MHD_SERVER_FLAGS_NONBLOCKING, http_port, NULL, NULL,
&ahc_request_handler, NULL, MHD_OPTION_END);
if (daemon == NULL) {
fprintf(stderr, "Failed to start HTTP daemon on port %d\n", http_port);
return 1;
}
printf("HTTP server started on port %d. Redis monitor thread running.\n", http_port);
// Keep the main thread alive
pthread_join(monitor_tid, NULL); // This will block indefinitely, or until thread exits
MHD_stop_daemon(daemon);
if (health_check_ctx) redisFree(health_check_ctx);
return 0;
}
Compile with pthreads and microhttpd: `gcc your_app.c -o your_app -lhiredis -lmicrohttpd -lpthread`
2. GCP Health Check Configuration:
In the GCP console, navigate to Compute Engine > Instance groups. Create or edit a Managed Instance Group. When configuring the health check:
- Type: HTTP
- Port: 8080 (or whatever port your C app listens on)
- Request Path: / (or the specific path your handler uses)
- Check Interval: e.g., 10 seconds
- Timeout: e.g., 5 seconds
- Healthy threshold: e.g., 2 consecutive successes
- Unhealthy threshold: e.g., 3 consecutive failures
When the health check fails for an instance (meaning your C app can’t reach Redis), the MIG will mark the instance as unhealthy. The MIG’s autohealing policy will then terminate the unhealthy instance and launch a new one. The new instance will start your C application, which will then attempt to connect to the *current* Redis master via Sentinel.
Deployment Considerations
Service Accounts and Permissions: Ensure the service account used by your Compute Engine instances has the necessary permissions to access GCP metadata if you choose that route for configuration. For Sentinel discovery, it’s often simpler to bake IPs into startup scripts or configuration files managed by Terraform/Ansible.
Firewall Rules: Meticulously configure GCP firewall rules to allow:
- Redis instances to communicate with each other (replication).
- Sentinel instances to communicate with Redis instances (monitoring).
- Sentinel instances to communicate with each other (quorum).
- Application instances to communicate with Sentinel instances (discovery).
- Application instances to communicate with the Redis master (data operations).
- External health checkers (GCP’s) to reach your application’s health check endpoint.
State Management: While Sentinel handles Redis failover, ensure your application’s own state (e.g., connection pools, session data not stored in Redis) is managed appropriately during restarts.
By combining Redis Sentinel for Redis HA, a resilient client application that queries Sentinel, and GCP Managed Instance Groups with health checks, you create a robust, automated failover system for your Redis deployments.