Large-Scale Search Platform Development for 2M+ Records

Project Overview

The platform managed a large volume of infrastructure and operational records used by registered teams for search, review, and export workflows. Its existing architecture made broad searches slow, large updates difficult, exports less dependable, and full database refreshes harder to control. The client needed a frontend that was easier to maintain for routine page work, along with stronger backend infrastructure for large dataset search, export processing, and database migration workflow. Maneuvrez rebuilt the system around a WordPress client-facing layer, a custom API server, PostgreSQL, Elasticsearch, Redis, RabbitMQ, Docker, and AWS, keeping the familiar user workflow while changing the architecture behind it.

The Challenge

The work focused on three principal issues that affected daily operations and platform confidence.

• Search performance declined when users worked with broad queries, heavier filters, or large result sets.
• Large CSV and Excel exports were unreliable or placed too much pressure on normal application requests.
• Refreshing data from the original Microsoft SQL Server source into PostgreSQL and Elasticsearch required a safer, repeatable process with validation and failure handling.

The Solution

Maneuvrez separated the client-facing website experience from the heavier backend search, export, and migration workload. WordPress gave the client a more manageable interface for page updates, while the custom API server handled the platform logic behind authenticated search forms, exports, migration events, and service coordination.

Search Architecture

Elasticsearch handled text and geolocation searches across the high-volume data platform. The search design supported large dataset search without making the relational database responsible for every query pattern. Stable sorting and pagination were added so users could move through large result windows with more consistent ordering, especially when records shared similar text, status, or location attributes.

Data and Fallback Layer

PostgreSQL served as the primary application database after data was moved from the Microsoft SQL Server source into the application environment. It supported the custom API server, structured records, and the fallback behaviour already required by the platform. Bulk import and structured database preparation helped make refreshes more manageable before records were indexed for search. When the search layer was unavailable, PostgreSQL could still support a reduced search path, helping avoid a complete break in user-facing search functionality.

Caching and Exports

Redis reduced repeated query work by caching search results until the next database refresh. Export processing supported large CSV and Excel files without tying the work directly to normal page requests. This helped keep user-facing activity more responsive while allowing registered users to download filtered results and receive clearer feedback when uploaded identifier files contained records that could not be matched.

Migration and Reliability

RabbitMQ coordinated background migration jobs so refreshes could be processed outside the request cycle. AWS S3 supported backup handling from the source database, while Docker improved service consistency across the deployment environment on AWS EC2. The refresh workflow included validation, controlled switching after successful migration, Elasticsearch re-indexing, cleanup, status updates, and safer failure recovery when an import or indexing step did not complete as expected. This gave the operations team a clearer process for regular refreshes without depending on direct source-database listeners.

Results

The rebuilt search and export platform delivered clearer operational reliability without relying on unsupported before-and-after metrics.

• Faster and more responsive searching across more than two million records
• Stable pagination and sorting for large result sets
• More dependable CSV and Excel exports
• A repeatable database refresh and migration process with safer recovery options

The architecture made the platform easier to operate, maintain, and extend. It also gave the client a more practical split between editable WordPress pages and the scalable backend system responsible for search, exports, caching, and migration logic. For business users, the change was most visible in a platform that behaved more predictably during everyday searches, downloads, and scheduled data refreshes.

Technology Stack

Related Maneuvrez Work

This project reflects Maneuvrez work in custom search platform development, backend APIs, migration workflows, and website development. Explore more services or review another enterprise search visibility project in the LuxuryProperty.com SEO case study. For teams evaluating large-scale search platform development, the main lesson is that search speed, export reliability, and data refresh control need to be designed together.