A global leader in the cruise line industry decided to address the limitations in its legacy reservation systems by developing a custom reservation system based on microservices architecture. This transition aimed to replace brand-specific reservation systems with a unified approach, leveraging the scalability and fault tolerance of microservices over monolithic design.
As part of this transformation, downstream systems needed to adopt an event-based message consumption pattern, facilitated by a real-time platform such as Amazon MSK.
Tredence played a crucial role in implementing a near real-time data pipeline, helping the client transition to an event-driven model. This shift not only made their pipeline real-time instead of batch-based but also enabled the client to derive actionable insights in near real-time, leveraging the finest data granularity with Amazon MSK.
Faced with maintaining multiple batch-oriented data pipelines due to their legacy reservation system, our client recognized the need to eliminate silos and consolidate into a unified model.
Their vision was to develop a unified, fault-tolerant, and scalable near real-time data ingestion pipeline from their microservices-based system to the ODS layer in Snowflake.
To achieve this transformation, Tredence partnered with the client to leverage Amazon MSK on AWS. Together, we built a fully dynamic, fault-tolerant, scalable, and configurable near real-time streaming pipeline.
Amazon MSK offers a fully managed service for building and running applications that use Apache Kafka to process streaming data.
The new pipeline aimed to leverage Amazon MSK's capabilities with the following core functionalities:
These enhancements not only modernized the pipeline to a truly distributed, fault-tolerant, and scalable design but also enabled the client to derive actionable insights from near real-time data. Additionally, the pipeline provided access to data at the finest event level for deeper analysis.
Understanding the need, Tredence designed and developed a dynamic, fault-tolerant, and fully scalable real-time ingestion framework. This framework established an end-to-end streaming ingestion pipeline, seamlessly connecting a microservices-based source application to an Operational Data Store (ODS) layer within Snowflake.
The solution harnessed the robust streaming capabilities of Amazon MSK, complemented by various other AWS services tailored to meet the pipeline's specific requirements.
Below is an architectural diagram illustrating the platform's solution.
Integrating our solution into the customer's ingestion landscape effectively addressed their key challenges:
The solution is primarily built around AWS services, with a few exceptions like the Schema Registry and Snowflake.
The following components were used to build the pipeline:
The introduction of the real-time ingestion framework into the client's ingestion landscape has enabled evolution from batch-oriented ingestion to real-time capability. This advancement enables actionable analytics in near real-time, a capability not achievable with the traditional batch-oriented approach.
The following high-value achievements are being offered:
A fully dynamic and configurable ready to use end-to-end streaming framework
Automated data consumption pipelines scale up
Low maintenance and operations cost
