Achieving Data Consistency in Microservice Architectures
Written on
Chapter 1: The Challenge of Data Consistency
In microservice architectures, the challenge of maintaining ACID transactions across multiple services is significant. While individual microservices can implement ACID transactions internally, coordinating these transactions across multiple services proves difficult. The SAGA design pattern presents a viable solution to this issue.
Consider an online retail platform that employs four microservices: payment, order, shipping, and stock management. When a customer places an order, the Order microservice sends out a request to check stock, payment status, and shipping availability.
If stock is available, payment is processed, and shipping is ready, everything proceeds smoothly. However, complications arise if stock runs out, payment fails, or shipping cannot be fulfilled. In this scenario, the Order microservice bears the burden of managing order status and the overall flow of the application. As your application scales, this could lead the Order microservice to evolve into a cumbersome monolith.
Even with synchronous communication, the core issues remain unchanged. The real challenge lies in defining responsibilities and ensuring data consistency.
For instance, when the Stock microservice receives a message indicating an order creation but discovers the product is sold out, the situation becomes problematic. If payment has already been processed and shipping scheduled, the Order microservice must now manage the entire workflow again, including reversing successful operations like payments and shipments. This complexity necessitates a well-crafted apology email for the customer.
Chapter 2: Implementing SAGA for Data Management
SAGA can be envisioned as a data management wrapper for microservices. This approach allows you to treat operations similarly to SQL transactions, where each stage must succeed for the entire operation to be committed. If any issue arises, the system rolls back the changes.
To utilize SAGA, adjustments to your API are necessary, introducing new components that increase complexity within your microservices architecture. With SAGA in charge, requests must originate from it rather than the Order microservice, allowing for better control over processes and outcomes.
Under SAGA, the application sends messages to indicate the success or failure of each internal transition within microservices. If all transitions are successful, a message confirming success is sent. Conversely, if an error occurs, the application must issue a rollback notification.
To manage multiple transitions simultaneously, a GUID can serve as a universal reference across microservices, ensuring that each published message includes this identifier.
Handling Undo Operations
Determining how to implement undo operations can be intricate, as it depends on your architecture, business logic, and policies.
- Hard Delete: This method removes all transaction records from the microservices. While it clears up memory, it also eliminates any trace of what transpired, even with logging.
- Soft Delete: This common approach modifies the transaction status within the microservice, keeping the data intact but hidden from end users.
- Undo Transition: This involves performing the opposite operation, such as issuing a refund instead of deleting payment data.
Timing the Response
SAGA governs the request and its delivery to microservices. Two main strategies exist for responding to the user:
- Response Upon Completion: This method ensures accuracy but may result in longer wait times for users.
- Immediate Response: This approach boosts availability, but the initial response might be misleading. For example, users might receive a message stating, "Order is created," followed by a later notification confirming the actual status after the SAGA transition concludes.
The first video titled "Using sagas to maintain data consistency in a microservice architecture" by Chris Richardson explains how SAGA design patterns can enhance data consistency across distributed systems.
The second video titled "Data Consistency Between Microservices" dives deeper into the challenges of maintaining data integrity in microservices environments.