Rebuilding a Large-Scale Search and Export Platform

Case Study: Rebuilding a Large-Scale Search and Export Platform for Faster, More Reliable Operations

Category: Custom Backend System / Data Platform Optimization / Search Infrastructure Industry: Infrastructure records and operational data Location: United States Status: Anonymized technical case study Scale: 2M+ searchable records Users: 150–200 registered operational users Core Stack: Python, PHP, JavaScript, CSS, WordPress, PostgreSQL, Microsoft SQL Server, Elasticsearch, Redis, RabbitMQ, Docker, AWS EC2, AWS S3

Hero Summary

A high-volume infrastructure records platform was struggling with slow searches, large result windows, unreliable exports, and a difficult update workflow. The original system had been built in Django, but the client needed a more manageable frontend experience for routine page updates and a stronger backend architecture for search, migration, caching, and export reliability.

The solution introduced a WordPress-based client-facing layer, a custom API server, PostgreSQL on AWS EC2, Elasticsearch for text and geo-location searches, Redis query caching, RabbitMQ-powered migration jobs, AWS S3 backup handling, and a safer full-database refresh workflow.

The result was a more reliable search and export system capable of handling 2M+ records, supporting 150–200 registered users, preserving stable result ordering, exporting data to CSV/Excel, and recovering more gracefully when search infrastructure was unavailable.

About the Platform

The platform served registered operational users who needed to search, filter, review, and export a large infrastructure-related records database. Users depended on accurate text-based search, location-based search, stable pagination, and downloadable results for their routine workflows.

The dataset was not static. The client maintained a local Microsoft SQL Server database and regularly refreshed the public/search-facing records, usually every month and sometimes after shorter intervals. Because the team was not authorized to implement direct update listeners or a child-sync database, the migration process had to safely handle complete refreshes rather than incremental record updates.

This created a difficult technical challenge: the platform had to make a large changing dataset searchable and exportable while keeping the update workflow reliable, repeatable, and recoverable.

The Challenge

The previous system created several operational issues.

• Large result windows could crash the site or make the search experience unreliable.

• Search performance was slow, especially when users worked with broader queries or heavier filters.

• Text-based and location-based searches were not accurate or flexible enough for the users' needs.

• Routine content and page updates were difficult because the client-facing site was not easy for non-technical users to maintain.

• The data update process was complex because the source database was refreshed regularly, but direct change listeners or incremental update tracking were not available.

• Initial direct imports were too slow for the volume of data being handled.

• Export workflows needed to support CSV/Excel output and provide useful error feedback when users uploaded incorrect identifiers.

• The system needed better failure handling so migration errors could be detected, reported, and resolved without leaving the platform in an unclear state.

For the business, this was more than a technical inconvenience. Search reliability affected the daily work of registered users. When searches were slow, inaccurate, or unstable, users lost time and confidence in the platform.

The Strategy

The strategy was to separate the client-facing website experience from the heavy backend search and migration workload.

The public/client-side layer was moved into WordPress so routine page changes could be handled more easily. Search forms were protected behind login and embedded through shortcodes, giving users a familiar interface while still allowing the backend to remain custom and performance-oriented.

The backend was redesigned around PostgreSQL, Elasticsearch, Redis, RabbitMQ, Docker, AWS EC2, AWS S3, and a custom API server. This allowed each part of the system to handle the job it was best suited for:

• WordPress handled client-facing pages, protected forms, and content flexibility.

• The custom API server handled search requests, exports, migrations, and communication between systems.

• PostgreSQL acted as the optimized relational data layer and fallback search layer.

• Elasticsearch handled fast text-based and geo-location searches.

• Redis cached query results until the next database refresh.

• RabbitMQ handled migration jobs asynchronously.

• AWS S3 stored source database backups for the migration pipeline.

• Docker improved deployment consistency for services.

What We Built

WordPress Client Layer

The client-facing side was rebuilt in WordPress to make small content and page updates easier to manage. Instead of requiring backend code changes for routine site edits, the client could work with a more familiar content management flow.

Search forms were login-protected and embedded using shortcodes. This allowed the frontend to stay flexible while still connecting to a custom backend API for the heavy search and export functionality.

The frontend included:

• Login-protected search forms
• Custom shortcode-based embedding
• Custom API calls
• Elementor-supported page editing
• User-facing export interactions
• Error feedback for uploaded files with incorrect identifiers

PostgreSQL Data Layer

PostgreSQL was introduced on AWS EC2 as a performance-optimized data layer for the search platform.

The first implementation used direct import, but it was too slow for the size and refresh pattern of the dataset. To improve this, the import process was redesigned using PostgreSQL bulk import with the `COPY` command.

Materialized views were added to support faster query patterns and a more stable search experience. PostgreSQL also acted as a fallback layer when Elasticsearch was unavailable, helping prevent search functionality from breaking entirely if the search index could not be reached.

Elasticsearch Search Layer

Elasticsearch was added for text-based and geo-location searches.

The index was built with proper mapping and indexing strategy so the platform could support more accurate search behavior. Because incremental update detection was not allowed and source data was regularly replaced, the system re-indexed Elasticsearch after each migration event.

The search layer was designed to support:

• Text-based searches
• Geo-location searches
• Large searchable datasets
• Stable result handling
• Fallback behavior through PostgreSQL

Redis Query Cache

Redis caching was added to reduce repeated query load and improve response behavior for common searches.

Search results were cached until the next database update. When the database was refreshed, cached query results could be invalidated so users were not served stale results after a new migration.

This helped streamline the user experience and reduce unnecessary repeated work across the backend services.

RabbitMQ Migration Workflow

A major part of the project was the migration pipeline.

The client maintained a local Microsoft SQL Server database. A Bash script created a backup of that database and stored it on AWS S3. After the backup was available, an API call was made to the custom API server as a new migration event.

RabbitMQ handled the migration request and started the backend workflow.

The migration process included:

1. Receive migration event through API call
2. Queue the migration request through RabbitMQ
3. Download the source backup from AWS S3
4. Create a temporary database
5. Import source data into PostgreSQL
6. Build/update materialized views
7. Re-index Elasticsearch with proper mappings
8. Switch the active database/search layer after successful migration
9. Remove temporary files
10. Mark migration as completed
11. Send success email
12. Send failure email if an error occurs

This queue-based workflow made the update process more controlled and easier to monitor.

Export System and Error Reporting

The platform supported CSV and Excel exports so registered users could download filtered search results.

The export workflow also handled uploaded files containing identifiers such as JPA or pole numbers. When users uploaded incorrect identifiers, the system generated an error list at the end in Excel format so users could see what failed and correct their input.

This improved the usefulness of exports by turning failed lookups into actionable feedback instead of silent errors.

Stable Sorting and Result Windows

The original platform struggled when result windows became large. The revised system added a reasonable result-window strategy and preserved sorting behavior for fields where similar text or status values could otherwise produce unstable ordering.

During import, index IDs were added to support stable ordering in cases where records had similar text/status fields. This made pagination and repeated searches more predictable for users.

Reliability and Failure Handling

Early migration attempts exposed failure cases. Proper exception handling was added so errors could be detected, migration status could be marked correctly, and success or failure emails could be sent.

This improved operational confidence because the team could see whether a refresh had completed successfully or failed during the process.

The system also avoided a single point of failure in the search experience by allowing PostgreSQL to act as a fallback when Elasticsearch was unavailable.

Results and Business Impact

The rebuilt system delivered a more stable and scalable foundation for a high-volume records platform.

The main improvements included:

• Better handling for 2M+ searchable records
• Faster and more accurate text-based search through Elasticsearch
• Improved geo-location search capability
• Redis caching for repeated query patterns
• PostgreSQL fallback when Elasticsearch was unavailable
• More reliable large result handling
• Stable pagination and ordering behavior
• CSV and Excel export support
• Error reporting for incorrect uploaded identifiers
• A queue-based migration workflow using RabbitMQ
• S3-backed database backup handling
• Full re-indexing after regular database refreshes
• Success and failure email notifications
• Easier client-side page updates through WordPress

The platform became easier to operate, easier to update, and more reliable for registered users who depended on search and export workflows.

What This Means for Businesses

Large datasets create business risk when search, export, and update workflows are not designed properly. A website may look simple from the outside, but behind the scenes it may need search indexing, caching, migration automation, failure handling, and fallback layers to keep users productive.

This project shows how Maneuvrez can design practical backend infrastructure around real business constraints. Instead of forcing the client into a completely new workflow, the solution connected the systems they already had with a more reliable search and migration architecture.

For businesses dealing with large records, internal portals, customer dashboards, operational databases, or complex search requirements, this kind of system can reduce manual work, improve reliability, and make data easier to use.

Looking Ahead

The next stage for a platform like this could include more advanced monitoring, migration dashboards, incremental sync support if permissions allow, enhanced analytics on user searches, stronger admin controls, and more automated validation before database refreshes.

The foundation already supports the most important requirement: a large operational dataset can be migrated, indexed, searched, cached, exported, and refreshed through a controlled workflow.

Call to Action

Need a Search or Data Platform That Does Not Break Under Real Usage?

Maneuvrez builds custom backend systems, search platforms, automation workflows, and high-performance web interfaces for businesses that need more than a basic website. If your current system is slow, hard to update, or unreliable under large datasets, we can help you plan a practical architecture that works around your real operational constraints.