WordPress Development Recipe: Staggered database writes for high-volume custom form fields using Rewrite API custom endpoints
Leveraging WordPress Rewrite API for Staggered Database Writes
When developing custom form solutions for WordPress that handle a high volume of submissions, particularly with numerous custom fields per submission, direct, synchronous database writes can become a performance bottleneck. This is especially true if the form processing involves complex validation, external API calls, or data enrichment before saving. This recipe outlines a strategy to decouple the initial form submission from the actual database write operation, improving perceived performance for the end-user and making the system more resilient to transient database issues. We’ll achieve this by utilizing WordPress’s Rewrite API to create custom endpoints that act as asynchronous processing queues.
Core Concept: The Asynchronous Processing Pipeline
The fundamental idea is to:
- Intercept the initial form submission via a standard POST request.
- Instead of processing and saving immediately, enqueue the submission data for background processing.
- Use a custom Rewrite API endpoint to trigger a background worker that processes these enqueued items.
- The worker then performs the actual database write operations.
This approach effectively creates a pipeline: submission -> enqueue -> background processing -> database write. This significantly reduces the latency of the initial user-facing request.
Step 1: Implementing the Form Submission Handler
We’ll start by creating a simple form and a PHP function to handle its submission. This function will not save data directly but will instead store it in a transient or a custom database table (for more robust persistence) to be picked up later. For simplicity in this example, we’ll use WordPress transients, which are suitable for short-term queuing.
Form HTML (Example)
This would typically be part of your plugin’s shortcode or template integration.
High Volume Form
PHP Submission Handler
This code should be placed in your plugin’s main file or an included file. It hooks into the `template_redirect` action to catch POST requests to our custom endpoint.
// In your plugin's main file or an included file
// Hook into template_redirect to catch our custom endpoint POST requests
add_action( 'template_redirect', 'my_high_volume_form_submission_handler' );
function my_high_volume_form_submission_handler() {
// Check if it's a POST request to our specific endpoint
if ( $_SERVER['REQUEST_METHOD'] === 'POST' && isset( $_SERVER['REQUEST_URI'] ) && strpos( $_SERVER['REQUEST_URI'], '/process-form-submission/' ) !== false ) {
// Verify nonce
if ( ! isset( $_POST['nonce_field'] ) || ! wp_verify_nonce( $_POST['nonce_field'], 'my_form_submission_nonce' ) ) {
wp_die( 'Security check failed!' );
}
// Sanitize and collect form data
$submission_data = array(
'field_one' => sanitize_text_field( $_POST['field_one'] ?? '' ),
'field_two' => sanitize_text_field( $_POST['field_two'] ?? '' ),
'field_three' => sanitize_textarea_field( $_POST['field_three'] ?? '' ),
'submitted_at' => current_time( 'mysql' ),
);
// Enqueue the submission data for background processing
// We'll use a transient as a simple queue. For high volume, a custom table or a dedicated queue system is better.
$queue_key = 'my_form_submission_queue';
$current_queue = get_transient( $queue_key );
if ( ! is_array( $current_queue ) ) {
$current_queue = array();
}
// Add the new submission to the queue
$current_queue[] = $submission_data;
// Store the updated queue back into the transient.
// Set an expiration time (e.g., 1 hour) to prevent indefinite storage if processing fails.
set_transient( $queue_key, $current_queue, HOUR_IN_SECONDS );
// Redirect back to the form page or a thank you page
// Use wp_redirect for proper redirection
wp_redirect( home_url( '/thank-you/' ) ); // Assuming a thank you page exists
exit;
}
}
Step 2: Registering the Custom Rewrite Endpoint
We need to tell WordPress about our custom endpoint so it can be recognized and routed correctly. This involves adding a rewrite rule and a query variable.
/**
* Add custom rewrite rule and query variable.
*/
function my_high_volume_form_rewrite_rules() {
// Add a query variable to identify our endpoint
add_rewrite_tag( '%process_form%', '([^/]+)/?$' ); // This is a placeholder, we'll use it to trigger our handler
// Add the rewrite rule: map /process-form-submission/ to a specific query
// The 'process_form=1' part will be our trigger.
add_rewrite_rule(
'^process-form-submission/?$',
'index.php?process_form=1',
'top'
);
}
add_action( 'init', 'my_high_volume_form_rewrite_rules' );
/**
* Add our custom query variable to WordPress.
*
* @param array $vars Existing query variables.
* @return array Modified query variables.
*/
function my_high_volume_form_query_vars( $vars ) {
$vars[] = 'process_form';
return $vars;
}
add_filter( 'query_vars', 'my_high_volume_form_query_vars' );
/**
* Flush rewrite rules on plugin activation/deactivation.
* This is crucial for the rewrite rules to take effect.
*/
function my_high_volume_form_activate() {
my_high_volume_form_rewrite_rules(); // Ensure rules are added
flush_rewrite_rules();
}
register_activation_hook( __FILE__, 'my_high_volume_form_activate' );
function my_high_volume_form_deactivate() {
flush_rewrite_rules(); // Flush rules on deactivation
}
register_deactivation_hook( __FILE__, 'my_high_volume_form_deactivate' );
After adding this code, you must visit the WordPress Permalinks settings page (Settings -> Permalinks) for the rewrite rules to be flushed and take effect. Alternatively, the activation/deactivation hooks will handle this when the plugin is activated/deactivated.
Step 3: Implementing the Background Processor
Now, we need a mechanism to process the items in our queue. This will be another custom endpoint, but this one will be triggered by a cron job or a scheduled task, not directly by user submission. For this example, we’ll simulate this by creating a separate function that can be called manually or via WP-Cron.
The Processing Function
/**
* Processes the form submission queue.
*/
function my_high_volume_form_process_queue() {
$queue_key = 'my_form_submission_queue';
$queue = get_transient( $queue_key );
if ( ! is_array( $queue ) || empty( $queue ) ) {
// No items in the queue
return;
}
// Process the first item in the queue
$submission_data = array_shift( $queue ); // Get and remove the first item
// --- Database Write Operation ---
// This is where you'd save the data to your custom post type,
// user meta, or a custom table.
// Example: Saving to a custom post type 'form_submissions'
$post_data = array(
'post_title' => 'Form Submission - ' . date('Y-m-d H:i:s'),
'post_content' => 'Field One: ' . $submission_data['field_one'] . "\n" .
'Field Two: ' . $submission_data['field_two'] . "\n" .
'Field Three: ' . $submission_data['field_three'],
'post_status' => 'publish',
'post_type' => 'form_submissions', // Ensure this post type is registered
'post_date' => $submission_data['submitted_at'],
'post_date_gmt' => $submission_data['submitted_at'],
);
$post_id = wp_insert_post( $post_data );
if ( ! is_wp_error( $post_id ) ) {
// Optionally add custom field data using update_post_meta
update_post_meta( $post_id, '_field_one', $submission_data['field_one'] );
update_post_meta( $post_id, '_field_two', $submission_data['field_two'] );
update_post_meta( $post_id, '_field_three', $submission_data['field_three'] );
update_post_meta( $post_id, '_submitted_at', $submission_data['submitted_at'] );
// Successfully processed, update the transient with the remaining queue
if ( empty( $queue ) ) {
delete_transient( $queue_key ); // Remove transient if queue is empty
} else {
set_transient( $queue_key, $queue, HOUR_IN_SECONDS ); // Update with remaining items
}
} else {
// Handle error: Log the error, re-queue the item, or move to an error queue.
// For simplicity, we'll just put it back at the front of the queue for retry.
array_unshift( $queue, $submission_data ); // Add back to the beginning
set_transient( $queue_key, $queue, HOUR_IN_SECONDS );
error_log( 'Error processing form submission: ' . $post_id->get_error_message() );
}
}
Triggering the Processor (WP-Cron)
We can use WP-Cron to schedule `my_high_volume_form_process_queue` to run periodically. This is a simulated cron; for true reliability, consider a server-level cron job that hits a specific URL.
/**
* Schedule the queue processing event.
*/
function my_high_volume_form_schedule_event() {
if ( ! wp_next_scheduled( 'my_high_volume_form_process_queue_event' ) ) {
// Schedule the event to run every 5 minutes
wp_schedule_event( time(), 'five_minutes', 'my_high_volume_form_process_queue_event' );
}
}
add_action( 'wp', 'my_high_volume_form_schedule_event' );
/**
* Hook into the scheduled event and run the processing function.
*/
function my_high_volume_form_run_scheduled_processing() {
my_high_volume_form_process_queue();
}
add_action( 'my_high_volume_form_process_queue_event', 'my_high_volume_form_run_scheduled_processing' );
/**
* Add a custom interval to WP Cron.
*
* @param array $schedules Existing schedules.
* @return array Modified schedules.
*/
function my_high_volume_form_add_cron_interval( $schedules ) {
$schedules['five_minutes'] = array(
'interval' => 300, // 5 minutes in seconds
'display' => __( 'Every 5 Minutes' ),
);
return $schedules;
}
add_filter( 'cron_schedules', 'my_high_volume_form_add_cron_interval' );
/**
* Clear the scheduled event on plugin deactivation.
*/
function my_high_volume_form_deactivate_schedule() {
$timestamp = wp_next_scheduled( 'my_high_volume_form_process_queue_event' );
if ( $timestamp ) {
wp_unschedule_event( $timestamp, 'my_high_volume_form_process_queue_event' );
}
}
register_deactivation_hook( __FILE__, 'my_high_volume_form_deactivate_schedule' );
With this setup, WP-Cron will periodically trigger `my_high_volume_form_process_queue`, which will pick up and process one item from the transient queue at a time. This prevents a single cron run from overwhelming the server with database writes.
Step 4: Enhancements and Considerations
1. Robust Queueing Mechanism
Transients are not ideal for very high volumes or long-term queuing. Consider:
- Custom Database Table: Create a dedicated table (e.g., `wp_form_submissions_queue`) to store submission data. This offers better control, indexing, and persistence. You’d then query this table for items to process.
- External Queue Services: For massive scale, integrate with services like Redis queues, RabbitMQ, or AWS SQS. This offloads queuing and processing entirely.
2. Error Handling and Retries
The current error handling is basic. Implement:
- Retry Logic: If a database write fails, instead of just re-queueing, implement exponential backoff for retries.
- Dead Letter Queue: After a certain number of failed attempts, move the submission to a “dead letter queue” for manual inspection.
- Logging: Comprehensive logging of successful and failed processing attempts is essential for debugging.
3. Security and Rate Limiting
The initial submission handler uses a nonce. For the background processing, ensure:
- Server-Level Cron: If using WP-Cron, it’s susceptible to being triggered by external requests. For critical processing, use a server-level cron job that hits a specific, secured URL (e.g., with an API key or IP restriction).
- Rate Limiting: Prevent abuse by implementing rate limiting on the initial submission endpoint.
4. Custom Post Type Registration
If you’re saving submissions as custom post types (as in the example), ensure you register it properly in your plugin:
/**
* Register the custom post type for form submissions.
*/
function my_high_volume_form_register_post_type() {
$labels = array(
'name' => _x( 'Form Submissions', 'Post type general name', 'your-text-domain' ),
'singular_name' => _x( 'Form Submission', 'Post type singular name', 'your-text-domain' ),
'menu_name' => _x( 'Form Submissions', 'Admin Menu text', 'your-text-domain' ),
'name_admin_bar' => _x( 'Form Submission', 'Add New on Toolbar', 'your-text-domain' ),
'add_new' => __( 'Add New', 'your-text-domain' ),
'add_new_item' => __( 'Add New Form Submission', 'your-text-domain' ),
'edit_item' => __( 'Edit Form Submission', 'your-text-domain' ),
'view_item' => __( 'View Form Submission', 'your-text-domain' ),
'all_items' => __( 'All Form Submissions', 'your-text-domain' ),
'search_items' => __( 'Search Form Submissions', 'your-text-domain' ),
'parent_item_colon' => __( 'Parent Form Submissions:', 'your-text-domain' ),
'not_found' => __( 'No form submissions found.', 'your-text-domain' ),
'not_found_in_trash' => __( 'No form submissions found in Trash.', 'your-text-domain' ),
'featured_image' => _x( 'Form Submission Cover Image', 'Overrides the "Featured Image" phrase for this post type. Added in 4.3', 'your-text-domain' ),
'set_featured_image' => _x( 'Set cover image', 'Overrides the "Set featured image" phrase for this post type. Added in 4.3', 'your-text-domain' ),
'remove_featured_image' => _x( 'Remove cover image', 'Overrides the "Remove featured image" phrase for this post type. Added in 4.3', 'your-text-domain' ),
'use_featured_image' => _x( 'Use as cover image', 'Overrides the "Use as featured image" phrase for this post type. Added in 4.3', 'your-text-domain' ),
'archives' => _x( 'Form Submission archives', 'The post type archive label used in nav menus. Default is "Post Archives". Added in 4.4', 'your-text-domain' ),
'insert_into_item' => _x( 'Insert into form submission', 'Overrides the "Insert into post." phrase (used when inserting media into a post). Added in 4.4', 'your-text-domain' ),
'uploaded_to_this_item' => _x( 'Uploaded to this form submission', 'Overrides the "Uploaded to this post" phrase (used when viewing media attached to a post). Added in 4.4', 'your-text-domain' ),
'filter_items_list' => _x( 'Filter form submissions list', 'Screen reader text for the filter links heading on the post type listing screen. Default is "Filter posts list". Added in 4.4', 'your-text-domain' ),
'items_list_navigation' => _x( 'Form submissions list navigation', 'Screen reader text for the pagination heading on the post type listing screen. Default is "Posts list navigation". Added in 4.4', 'your-text-domain' ),
'items_list' => _x( 'Form submissions list', 'Screen reader text for the items list heading on the post type listing screen. Default is "Posts list". Added in 4.4', 'your-text-domain' ),
);
$args = array(
'labels' => $labels,
'public' => true,
'publicly_queryable' => true,
'show_ui' => true,
'show_in_menu' => true,
'query_var' => true,
'rewrite' => array( 'slug' => 'submission' ),
'capability_type' => 'post',
'has_archive' => 'submissions',
'hierarchical' => false,
'menu_position' => null,
'supports' => array( 'title', 'editor', 'custom-fields' ), // 'custom-fields' is important if you want to use them directly
'show_in_rest' => true, // For Gutenberg editor compatibility
);
register_post_type( 'form_submissions', $args );
}
add_action( 'init', 'my_high_volume_form_register_post_type' );
5. Performance Tuning
For very high volumes, consider batching writes. Instead of processing one item at a time in the cron job, process a small batch (e.g., 5-10 items) per cron run. This can be more efficient for database operations but requires careful management of the queue and error handling.
Conclusion
By decoupling form submissions from immediate database writes using WordPress’s Rewrite API and a background processing queue, you can significantly improve the performance and scalability of your custom form solutions. This recipe provides a foundational pattern that can be extended with more robust queueing mechanisms and error handling for production environments dealing with high submission volumes.