Server Monitoring Best Practices: Keeping Your PHP App and Redis Clusters Alive on OVH
Proactive Redis Cluster Health Checks with `redis-cli` and Custom Scripts
Maintaining the health of a Redis cluster, especially in a production environment hosted on OVH, requires more than just basic uptime checks. We need to monitor key performance indicators (KPIs) that directly impact application responsiveness and data integrity. This involves deep dives into Redis’s internal state, not just external connectivity.
A fundamental tool for this is the `redis-cli`. Beyond simple `PING` commands, we can leverage its capabilities to inspect cluster status, memory usage, and replication health. For automated monitoring, scripting these checks is essential. We’ll focus on a Python script that orchestrates these `redis-cli` commands and interprets their output.
Cluster State and Node Reachability
The first line of defense is ensuring all nodes in the cluster are aware of each other and are in a stable state. The `CLUSTER INFO` command provides a wealth of information, but `CLUSTER NODES` is particularly useful for a quick overview of node status.
Here’s how you can extract and parse this information using `redis-cli` and a Python script. We’ll assume your Redis nodes are accessible via a list of host:port combinations. For simplicity, this example connects to one node and assumes it can reach others via the cluster bus.
Python Script for Cluster Node Status
This script connects to a specified Redis node, executes `CLUSTER NODES`, and then iterates through the output to check the status of each node. It flags nodes that are not `master` or `slave` (indicating potential issues) or those that are marked as `fail`.
import redis
import subprocess
import sys
def check_redis_cluster_nodes(host='localhost', port=6379, password=None):
"""
Checks the status of Redis cluster nodes.
Returns a dictionary of node statuses and a list of problematic nodes.
"""
node_statuses = {}
problematic_nodes = []
try:
# Connect to a single node to get cluster info
r = redis.StrictRedis(host=host, port=port, password=password, decode_responses=True)
r.ping() # Ensure connection is valid
# Execute CLUSTER NODES command
# Using subprocess to ensure we get the raw output from redis-cli
cmd = ['redis-cli', '-h', host, '-p', str(port)]
if password:
cmd.extend(['-a', password])
cmd.append('CLUSTER')
cmd.append('NODES')
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = process.communicate()
if process.returncode != 0:
print(f"Error executing redis-cli CLUSTER NODES: {stderr.decode()}", file=sys.stderr)
return {}, ["redis-cli execution error"]
output = stdout.decode()
for line in output.splitlines():
parts = line.split()
if not parts:
continue
node_id = parts[0]
ip_port = parts[1].split('@')[0] # Get IP:Port, ignore bus address
flags = parts[2]
master_id = parts[3]
ping_sent = parts[4]
ping_recv = parts[5]
epoch = parts[6]
status = "unknown"
if 'master' in flags:
status = "master"
elif 'slave' in flags:
status = "slave"
node_info = {
'id': node_id,
'ip_port': ip_port,
'flags': flags,
'master_id': master_id,
'ping_sent': ping_sent,
'ping_recv': ping_recv,
'epoch': epoch,
'status': status
}
node_statuses[ip_port] = node_info
if 'fail' in flags:
problematic_nodes.append(f"{ip_port} (ID: {node_id}) is marked as FAIL")
if status == "unknown" and node_id != 'myself': # Exclude the current node if it's not explicitly master/slave in this output
problematic_nodes.append(f"{ip_port} (ID: {node_id}) has an unexpected flag status: {flags}")
# Additional check: ensure all masters have slaves (if applicable)
masters = {ip: info for ip, info in node_statuses.items() if info['status'] == 'master'}
slaves = {ip: info for ip, info in node_statuses.items() if info['status'] == 'slave'}
for master_ip, master_info in masters.items():
has_slave = False
for slave_ip, slave_info in slaves.items():
if slave_info['master_id'] == master_info['id']:
has_slave = True
break
# This check might be too strict depending on your cluster setup (e.g., single master)
# if not has_slave and len(masters) > 1: # Only flag if there are multiple masters and this one has no slaves
# problematic_nodes.append(f"Master {master_ip} (ID: {master_info['id']}) has no slaves.")
except redis.exceptions.ConnectionError as e:
print(f"Could not connect to Redis at {host}:{port}: {e}", file=sys.stderr)
return {}, [f"Connection error to {host}:{port}"]
except Exception as e:
print(f"An unexpected error occurred: {e}", file=sys.stderr)
return {}, [f"Unexpected error: {e}"]
return node_statuses, problematic_nodes
if __name__ == "__main__":
# Example usage: Replace with your OVH Redis cluster details
# For a production setup, these would come from environment variables or a config file.
REDIS_HOST = 'your_redis_master_node_ip' # e.g., '192.168.1.10'
REDIS_PORT = 6379
REDIS_PASSWORD = 'your_redis_password' # Set to None if no password
print(f"Checking Redis cluster status for {REDIS_HOST}:{REDIS_PORT}...")
statuses, problems = check_redis_cluster_nodes(REDIS_HOST, REDIS_PORT, REDIS_PASSWORD)
if not statuses and problems:
print("\n--- CLUSTER STATUS: CRITICAL ---")
for problem in problems:
print(f"- {problem}")
sys.exit(1) # Exit with error code for monitoring systems
print("\n--- CLUSTER NODE STATUS ---")
for ip_port, info in statuses.items():
print(f"Node: {ip_port} (ID: {info['id']})")
print(f" Flags: {info['flags']}")
print(f" Master ID: {info['master_id']}")
print(f" Status: {info['status']}")
print(f" Ping Sent: {info['ping_sent']}, Ping Received: {info['ping_recv']}")
print(f" Epoch: {info['epoch']}")
if problems:
print("\n--- POTENTIAL ISSUES DETECTED ---")
for problem in problems:
print(f"- {problem}")
sys.exit(1) # Exit with error code for monitoring systems
else:
print("\n--- CLUSTER STATUS: OK ---")
sys.exit(0)
Memory Usage and Eviction Policies
High memory usage is a common precursor to performance degradation and data loss (if eviction is not configured correctly). We need to monitor `used_memory` and `used_memory_peak`, and understand the `maxmemory` setting and the configured `maxmemory-policy`.
The `INFO memory` command provides these crucial metrics. We can parse this output to trigger alerts when memory usage exceeds predefined thresholds.
Python Script for Memory Monitoring
import redis
import sys
def check_redis_memory(host='localhost', port=6379, password=None, memory_threshold_percent=85, peak_memory_threshold_percent=95):
"""
Checks Redis memory usage against thresholds.
Returns a dictionary of memory stats and a list of alerts.
"""
memory_stats = {}
alerts = []
try:
r = redis.StrictRedis(host=host, port=port, password=password, decode_responses=True)
r.ping()
info_memory = r.info('memory')
memory_stats['used_memory_human'] = info_memory.get('used_memory_human')
memory_stats['used_memory'] = int(info_memory.get('used_memory', 0))
memory_stats['used_memory_peak_human'] = info_memory.get('used_memory_peak_human')
memory_stats['used_memory_peak'] = int(info_memory.get('used_memory_peak', 0))
memory_stats['maxmemory_human'] = info_memory.get('maxmemory_human')
memory_stats['maxmemory'] = int(info_memory.get('maxmemory', 0))
memory_stats['maxmemory_policy'] = info_memory.get('maxmemory_policy', 'noeviction')
if memory_stats['maxmemory'] > 0:
used_percent = (memory_stats['used_memory'] / memory_stats['maxmemory']) * 100
peak_used_percent = (memory_stats['used_memory_peak'] / memory_stats['maxmemory']) * 100
memory_stats['used_percent'] = round(used_percent, 2)
memory_stats['peak_used_percent'] = round(peak_used_percent, 2)
if used_percent > memory_threshold_percent:
alerts.append(f"Current memory usage ({memory_stats['used_memory_human']}) is {used_percent:.2f}% of maxmemory ({memory_stats['maxmemory_human']}). Threshold: {memory_threshold_percent}%. Policy: {memory_stats['maxmemory_policy']}")
if peak_used_percent > peak_memory_threshold_percent:
alerts.append(f"Peak memory usage ({memory_stats['used_memory_peak_human']}) is {peak_used_percent:.2f}% of maxmemory ({memory_stats['maxmemory_human']}). Threshold: {peak_memory_threshold_percent}%. Policy: {memory_stats['maxmemory_policy']}")
else:
# maxmemory is 0, meaning no limit is set. This might be intentional or an oversight.
# We can still log current and peak usage.
memory_stats['used_percent'] = None
memory_stats['peak_used_percent'] = None
alerts.append(f"Maxmemory is not set (0). Current usage: {memory_stats['used_memory_human']}, Peak usage: {memory_stats['used_memory_peak_human']}. Consider setting maxmemory and a policy.")
except redis.exceptions.ConnectionError as e:
print(f"Could not connect to Redis at {host}:{port}: {e}", file=sys.stderr)
return {}, [f"Connection error to {host}:{port}"]
except Exception as e:
print(f"An unexpected error occurred: {e}", file=sys.stderr)
return {}, [f"Unexpected error: {e}"]
return memory_stats, alerts
if __name__ == "__main__":
REDIS_HOST = 'your_redis_master_node_ip'
REDIS_PORT = 6379
REDIS_PASSWORD = 'your_redis_password' # Set to None if no password
# Thresholds in percentage of maxmemory
CURRENT_MEMORY_THRESHOLD = 80
PEAK_MEMORY_THRESHOLD = 90
print(f"Checking Redis memory usage for {REDIS_HOST}:{REDIS_PORT}...")
memory_info, memory_alerts = check_redis_memory(
REDIS_HOST, REDIS_PORT, REDIS_PASSWORD,
memory_threshold_percent=CURRENT_MEMORY_THRESHOLD,
peak_memory_threshold_percent=PEAK_MEMORY_THRESHOLD
)
if not memory_info and memory_alerts:
print("\n--- MEMORY STATUS: CRITICAL ---")
for alert in memory_alerts:
print(f"- {alert}")
sys.exit(1)
print("\n--- REDIS MEMORY USAGE ---")
print(f" Current Usage: {memory_info.get('used_memory_human', 'N/A')}")
print(f" Peak Usage: {memory_info.get('used_memory_peak_human', 'N/A')}")
print(f" Max Memory: {memory_info.get('maxmemory_human', 'N/A')}")
print(f" Max Memory Policy: {memory_info.get('maxmemory_policy', 'N/A')}")
if memory_info.get('used_percent') is not None:
print(f" Current Usage % of Max: {memory_info.get('used_percent', 'N/A')}%")
if memory_info.get('peak_used_percent') is not None:
print(f" Peak Usage % of Max: {memory_info.get('peak_used_percent', 'N/A')}%")
if memory_alerts:
print("\n--- MEMORY ALERTS ---")
for alert in memory_alerts:
print(f"- {alert}")
sys.exit(1)
else:
print("\n--- MEMORY STATUS: OK ---")
sys.exit(0)
Replication Lag and Health
In a Redis cluster, especially when using Sentinel for high availability or a master-replica setup, replication lag is critical. Significant lag can lead to stale data being served by replicas, impacting application consistency. The `INFO replication` command provides `master_repl_offset` and `slave_repl_offset` which we can use to calculate lag.
Python Script for Replication Lag
import redis
import sys
def check_redis_replication_lag(host='localhost', port=6379, password=None, lag_threshold_seconds=60):
"""
Checks Redis replication lag.
Returns a dictionary of replication stats and a list of alerts.
"""
replication_stats = {}
alerts = []
try:
r = redis.StrictRedis(host=host, port=port, password=password, decode_responses=True)
r.ping()
info_replication = r.info('replication')
replication_stats['role'] = info_replication.get('role')
if replication_stats['role'] == 'master':
replication_stats['master_repl_offset'] = int(info_replication.get('master_repl_offset', 0))
replication_stats['connected_slaves'] = int(info_replication.get('connected_slaves', 0))
replication_stats['slave_count'] = replication_stats['connected_slaves']
slaves_info = []
for i in range(replication_stats['connected_slaves']):
slave_key = f'slave{i}'
slave_data = info_replication.get(slave_key, {})
if isinstance(slave_data, str): # redis-py might return string if only one slave
slave_data = dict(item.split(":") for item in slave_data.split(","))
slave_ip_port = slave_data.get('ip', 'N/A') + ':' + slave_data.get('port', 'N/A')
slave_offset = int(slave_data.get('offset', 0))
slave_lag_time = int(slave_data.get('lag', 0)) # This is often reported in seconds by redis-cli INFO
slaves_info.append({
'ip_port': slave_ip_port,
'offset': slave_offset,
'lag_seconds': slave_lag_time
})
if slave_lag_time > lag_threshold_seconds:
alerts.append(f"Slave {slave_ip_port} is lagging by {slave_lag_time} seconds (threshold: {lag_threshold_seconds}s).")
replication_stats['slaves'] = slaves_info
elif replication_stats['role'] == 'slave':
replication_stats['master_host'] = info_replication.get('master_host', 'N/A')
replication_stats['master_port'] = int(info_replication.get('master_port', 0))
replication_stats['master_repl_offset'] = int(info_replication.get('master_repl_offset', 0))
replication_stats['slave_repl_offset'] = int(info_replication.get('slave_repl_offset', 0))
# Calculate lag based on offsets if possible, but 'lag' field is more direct if available
# The 'lag' field in INFO replication for a slave is usually the reported lag from the master.
# If not, we'd need to compare master_repl_offset and slave_repl_offset, which is less precise without timestamps.
# The 'lag' field is generally preferred.
replication_stats['lag_seconds'] = int(info_replication.get('lag', 0))
if replication_stats['lag_seconds'] > lag_threshold_seconds:
alerts.append(f"This slave is lagging by {replication_stats['lag_seconds']} seconds (threshold: {lag_threshold_seconds}s). Master: {replication_stats['master_host']}:{replication_stats['master_port']}")
else:
alerts.append(f"Unknown Redis role: {replication_stats['role']}")
except redis.exceptions.ConnectionError as e:
print(f"Could not connect to Redis at {host}:{port}: {e}", file=sys.stderr)
return {}, [f"Connection error to {host}:{port}"]
except Exception as e:
print(f"An unexpected error occurred: {e}", file=sys.stderr)
return {}, [f"Unexpected error: {e}"]
return replication_stats, alerts
if __name__ == "__main__":
REDIS_HOST = 'your_redis_node_ip' # Can be master or slave
REDIS_PORT = 6379
REDIS_PASSWORD = 'your_redis_password' # Set to None if no password
REPLICATION_LAG_THRESHOLD = 30 # seconds
print(f"Checking Redis replication lag for {REDIS_HOST}:{REDIS_PORT}...")
rep_info, rep_alerts = check_redis_replication_lag(
REDIS_HOST, REDIS_PORT, REDIS_PASSWORD,
lag_threshold_seconds=REPLICATION_LAG_THRESHOLD
)
if not rep_info and rep_alerts:
print("\n--- REPLICATION STATUS: CRITICAL ---")
for alert in rep_alerts:
print(f"- {alert}")
sys.exit(1)
print("\n--- REDIS REPLICATION STATUS ---")
print(f" Role: {rep_info.get('role', 'N/A')}")
if rep_info.get('role') == 'master':
print(f" Master Repl Offset: {rep_info.get('master_repl_offset', 'N/A')}")
print(f" Connected Slaves: {rep_info.get('slave_count', 'N/A')}")
if rep_info.get('slaves'):
print(" Slaves:")
for slave in rep_info['slaves']:
print(f" - {slave['ip_port']}: Offset={slave['offset']}, Lag={slave['lag_seconds']}s")
elif rep_info.get('role') == 'slave':
print(f" Master Host: {rep_info.get('master_host', 'N/A')}:{rep_info.get('master_port', 'N/A')}")
print(f" Master Repl Offset: {rep_info.get('master_repl_offset', 'N/A')}")
print(f" Slave Repl Offset: {rep_info.get('slave_repl_offset', 'N/A')}")
print(f" Lag: {rep_info.get('lag_seconds', 'N/A')}s")
if rep_alerts:
print("\n--- REPLICATION ALERTS ---")
for alert in rep_alerts:
print(f"- {alert}")
sys.exit(1)
else:
print("\n--- REPLICATION STATUS: OK ---")
sys.exit(0)
PHP Application Monitoring: Beyond Basic Uptime
For PHP applications, especially those relying on Redis for caching, session management, or as a message broker, monitoring needs to be granular. We’ll look at integrating application-level metrics and error tracking.
Application Performance Monitoring (APM) with Prometheus and PHP Exporter
Prometheus is a de facto standard for metrics collection in modern infrastructure. To expose PHP application metrics, we can use libraries like `prometheus_client` for PHP. This allows us to instrument our code to expose custom metrics such as request latency, cache hit/miss ratios, and queue depths.
Instrumenting PHP Code
First, install the `prometheus_client` library. Using Composer is the standard approach:
composer require prometheus_client/prometheus_client
Then, in your PHP application, initialize the client and define your metrics. A common pattern is to have a dedicated endpoint (e.g., `/metrics`) that the Prometheus server scrapes.
<?php
require __DIR__ . '/vendor/autoload.php';
use Prometheus\CollectorRegistry;
use Prometheus\RenderTextFormat;
use Prometheus\Storage\InMemory; // Or Redis, APCu, etc. for persistence
// Initialize the registry (use Redis for production to persist metrics across requests)
// For simplicity, we'll use InMemory here. For production, consider:
// use Prometheus\Storage\Redis as RedisStorage;
// $redis = new Redis(); $redis->connect('127.0.0.1', 6379);
// $registry = new CollectorRegistry(new RedisStorage($redis));
$registry = new CollectorRegistry(new InMemory());
// Define metrics
// Counter for total requests
$requestCounter = $registry->registerCounter(
'myapp', // Namespace
'requests_total', // Metric name
'Total number of HTTP requests', // Help text
['method', 'endpoint'] // Labels
);
// Gauge for current active connections (example)
$activeConnections = $registry->registerGauge(
'myapp',
'active_connections',
'Number of active connections',
['type']
);
// Histogram for request latency
$requestLatency = $registry->registerHistogram(
'myapp',
'request_latency_seconds',
'HTTP request latency in seconds',
['method', 'endpoint'],
[0.005, 0.01, 0.025, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10] // Buckets
);
// --- Example Usage within your application ---
// In your request handling logic:
// Assume $requestMethod, $requestEndpoint are determined
// $requestMethod = $_SERVER['REQUEST_METHOD'];
// $requestEndpoint = $_SERVER['REQUEST_URI']; // Simplified
// Increment request counter
// $requestCounter->incBy(1, [$requestMethod, $requestEndpoint]);
// Record request latency
// $startTime = microtime(true);
// ... process request ...
// $duration = microtime(true) - $startTime;
// $requestLatency->observe($duration, [$requestMethod, $requestEndpoint]);
// Update active connections gauge
// $activeConnections->set(getCurrentUserCount(), ['web']);
// --- Metrics Endpoint ---
// This part would typically be handled by a router or a dedicated script.
// If this script is accessed via /metrics:
if (isset($_SERVER['REQUEST_URI']) && $_SERVER['REQUEST_URI'] === '/metrics') {
header('Content-Type: text/plain; version=0.0.4');
$renderer = new RenderTextFormat();
echo $renderer->toHttpBody($registry);
exit;
}
// --- Example of Redis interaction metric ---
// Assuming you have a Redis client instance $redisClient
// $redisCacheHits = $registry->registerCounter('myapp', 'redis_cache_hits_total', 'Total Redis cache hits', ['key_prefix']);
// $redisCacheMisses = $registry->registerCounter('myapp', 'redis_cache_misses_total', 'Total Redis cache misses', ['key_prefix']);
// function getFromCache($key, $redisClient, $ttl = 3600) {
// $value = $redisClient->get($key);
// if ($value === false) {
// $redisCacheMisses->inc(['user_data']); // Example label
// // ... fetch data ...
// $redisClient->setex($key, $ttl, $fetchedValue);
// return $fetchedValue;
// } else {
// $redisCacheHits->inc(['user_data']); // Example label
// return $value;
// }
// }
?>
<!-- Your main application logic would go here -->
<h1>Welcome to the App!</h1>
<p>Metrics are available at /metrics</p>
With this setup, your Prometheus server can scrape the `/metrics` endpoint of your PHP application. You can then build dashboards in Grafana to visualize these metrics, correlate them with Redis performance, and set up alerting rules.
Error Tracking and Logging
Robust error tracking is non-negotiable. For PHP, integrating with services like Sentry, Bugsnag, or even a self-hosted solution like Graylog or ELK stack is crucial. This involves:
- Structured Logging: Ensure your application logs are in a machine-readable format (e.g., JSON). This makes parsing and analysis by log aggregation tools much easier.
- Error Reporting SDKs: Use official SDKs for your chosen error tracking service. These SDKs often capture stack traces, request context, user information, and environment details automatically.
- Centralized Log Aggregation: Ship your application logs (and server logs) to a central location.
Example: JSON Logging with Monolog
Monolog is a popular logging library for PHP. We can configure it to output JSON, which is ideal for log aggregators.
<?php
require __DIR__ . '/vendor/autoload.php';
use Monolog\Logger;
use Monolog\Handler\StreamHandler;
use Monolog\Formatter\JsonFormatter;
// Create a logger instance
$logger = new Logger('my_app_logger');
// Create a stream handler
$streamHandler = new StreamHandler(__DIR__ . '/logs/app.log', Logger::DEBUG);
// Set the formatter to JSONFormatter
$streamHandler->setFormatter(new JsonFormatter());
// Add the handler to the logger
$logger->pushHandler($streamHandler);
// --- Example Usage ---
// Log an informational message
$logger->info('User logged in', ['user_id' => 123, 'ip_address' => '192.168.1.100']);
// Log an error
try {
// Simulate an error, e.g., Redis connection failure
throw new \RedisException('Failed to connect to Redis server.');
} catch (\RedisException $e) {
$logger->error('Redis operation failed', [
'message' => $e->getMessage(),
'code' => $e->getCode(),
'file' => $e->getFile(),
'line' => $e->getLine(),
'trace' => $e->getTraceAsString() // Be cautious with large traces in logs
]);
}
// Log a warning
$logger->warning('High Redis memory usage detected', ['current_mb' => 950, 'max_mb' => 1000]);
?>
When this script runs, the `app.log` file will contain JSON objects, making it easy to ingest into systems like Elasticsearch or Splunk. For error reporting specifically, integrating an SDK like Sentry would look like this:
<?php require __DIR__ . '/vendor/autoload.php'; // Assuming Sentry SDK is installed via Composer // require 'sentry/sentry-php/src/Autoloader.php'; // \Sentry\Autoloader::register(); // Initialize Sentry SDK // \Sentry\init([ // 'dsn' => 'YOUR_SENTRY_DSN', // 'environment' => 'production', // 'release' => '[email protected]', // ]); // Example of capturing an exception // try { // // ... code that might throw an exception ... // throw new \Exception("Something went wrong in the application logic."); // } catch (\Exception $e) { // // Sentry SDK automatically captures uncaught exceptions if configured // // For explicit capture: // // \Sentry\captureException($e); // // // Log to Monolog as well for structured logging // // $logger->error('Application exception', ['message' => $e->getMessage()]); // } ?>
OVH Infrastructure Monitoring Integration
OVH provides its own set of monitoring tools and APIs. Integrating these with your custom application and Redis monitoring is key to a holistic view.
Leveraging OVH API for Instance Metrics
OVH’s Public Cloud API allows you to programmatically access metrics for your instances, such as CPU utilization, network traffic, and disk I/O. You can use these metrics to:
- Correlate application performance issues with underlying infrastructure load.
- Set up alerts for resource exhaustion on your OVH instances.
- Automate scaling decisions based on infrastructure metrics.
You can write scripts (e.g., in Python using the `ovh` SDK) to fetch these metrics and push them into your central monitoring system (like Prometheus) or trigger alerts directly.
Example: Fetching OVH Instance CPU Usage (Conceptual)
# This is a conceptual example. Actual API calls and SDK usage may vary.
# You'll need to install the 'ovh' Python SDK: pip install ovh
import ovh
import datetime
# Initialize OVH client
# Ensure you have configured your credentials (e.g., via environment variables or a config file)
# client = ovh.Client(endpoint='ovh-eu') # Or your specific region
def get_ovh_instance_metrics(instance_id, metric_name='cpu.total', period_seconds=300):
"""
Fetches a specific metric for an OVH instance.
"""
# Placeholder for actual API call
# This requires proper authentication and knowledge of the OVH API structure for metrics.
# Example structure:
# now = datetime.datetime.utcnow()
# since = now - datetime.timedelta(seconds=period_seconds)
#
# try:
# metrics = client.get(f'/cloud/project/YOUR_PROJECT_ID/metrics/{instance_id}/{metric_name}',
# fromTime=int(since.timestamp()), toTime=int(now.timestamp()),
# interval=60) # e.g., 60 seconds interval