As applications scale and demand increases, developers often look to horizontally scale their databases to handle larger workloads. While this approach helps with performance and availability, it introduces a less visible but critical challenge: managing transactions across multiple nodes.
In a traditional, single-node relational database, maintaining ACID properties is straightforward, everything happens in one place. But when data and operations span across distributed systems, transactions become complex, fragile, and costly to orchestrate.
This post helps you to understand why distributed transactions are difficult, how they impact system performance under heavy load, and what tools or architectural patterns you might consider to overcome these limitations.
1. Horizontal Scaling
Horizontal scaling refers to adding more machines (nodes) to a system to distribute the load across multiple computing resources. Each node runs an instance of the application or database and participates in processing requests.
In the context of Databases,
Vertical Scaling: Increases capacity of a single database instance by upgrading hardware (CPU, RAM, IOPS), so it can serve more traffic/user requests. It is simpler to implement, and minimal changes might be needed to the application code or DBMS configuration.
But Vertical Scaling is subject to physical hardware constraints. High availability may still require replicas, adding more hardware/procuring high end servers always incurs cost.
Horizontal Scaling: It involves distributing data and queries across multiple database instances. Following are the some common approaches used widely.
· Read Replicas: multiple read-only replicas handle read queries, and the primary node handles writes.
· Sharding: dataset is partitioned by shard key (e.g., user ID ranges), each shard resides on a different node.
· Federated Databases: A federated database system integrates several independent, often geographically distributed, databases into one virtual database. This unified view allows users and applications to query data as if it resided in a single database, without needing to be aware of the underlying complexities or physical locations of the individual databases. In this model, each node is responsible for a subset of data, with coordination at the application layer.
1.1 Technical Challenges of Horizontal Scaling
Data Partitioning (Sharding): requires a shard key strategy that minimizes cross-shard queries. If the shard key is not desingned properly, then there is a risk of uneven data distribution (hot spots).
Distributed Transactions: ACID properties guarantees across multiple nodes require protocols like 2PC (Two-Phase Commit) or saga patterns, which add latency and complexity.
Data Consistency: Achieving strong consistency across replicas or shards is non-trivial. Here CAP Theorem applies, which states that consistency, availability, and partition tolerance cannot be fully achieved simultaneously in distributed systems.
Replication Lag: In asynchronous replication, data changes made to the primary database are not immediately reflected in the read replicas. As a result, applications reading from replicas might receive stale or outdated data. This can lead to inconsistencies, especially in systems that rely on real-time accuracy.
Joins are Expensive: When data is spread across multiple shards, performing operations like joins becomes challenging. Joins across shards are either very expensive in terms of performance or not supported natively by many databases. This limitation can slow down complex queries and affect the overall efficiency of the system.
To handle these challenges, applications often require extra logic at the application level or need to use distributed query engines. These tools help coordinate data across shards and replicas, but they also add complexity to the system architecture.
2. Distributed Transactions
A distributed transaction is a database transaction that spans multiple independent systems or databases (usually across different servers or nodes), and must be executed in a coordinated way so that either all operations succeed, or all fail, and ensures atomicity and consistency across systems.
2.1 ACID Transactions in a Single Node vs Distributed Environment
In a non-distributed relational database, all components reside within a single system. Write operations are typically handled by a primary node, while read operations can be offloaded to multiple replica (or "slave") nodes. The database engine centrally manages tables, indexes, logs, buffers, and locks.
When a transaction spans multiple tables, it is coordinated internally using local ACID properties there is no need for external coordination.
BEGIN;
UPDATE accounts SET balance = balance - 500 WHERE id = 1;
UPDATE accounts SET balance = balance + 500 WHERE id = 2;
COMMIT;
This entire transaction is atomic and isolated because it is executed within a single database engine on one machine.
Whereas in a distributed environment:
· Atomicity: Every part of the transaction must either commit completely or not at all, spanning multiple nodes.
· Consistency: All nodes must maintain a consistent state (Some systems support Eventual consistency), even in the event of failures.
· Isolation: Concurrent transactions must be properly managed across distributed resources.
· Durability: Once committed, a transaction’s changes must be permanently recorded across multiple systems.
2.2 Why it is difficult to implement Distributed Transactions?
Network Latency and Failures: Implementing distributed transactions is difficult largely because they depend heavily on network communication between multiple, physically separate nodes. Unlike a single-server system where data and transaction coordination happen internally with minimal delay, distributed systems must send messages back and forth over a network. This introduces network latency, meaning every step in the transaction like preparing, committing, or rolling back changes on different nodes takes longer simply because of the time it takes for data to travel across the network. Moreover, networks are inherently unreliable, packets can be lost, delayed, duplicated, or nodes can become temporarily unreachable due to hardware failures, maintenance, or congestion. These uncertainties raise the risk of partial failures where some nodes successfully commit their part of the transaction, while others do not. Handling such inconsistencies requires complex coordination protocols and recovery mechanisms, which add overhead and further increase latency. Together, network delays and the possibility of failures make maintaining atomicity and consistency across distributed transactions a challenging problem in distributed systems.
Two-Phase Commit (2PC): It is a classic protocol designed to ensure that distributed transactions maintain ACID properties (atomicity, consistency, isolation, and durability) across multiple database nodes. It works in two main stages:
· First, in the prepare phase, the coordinator sends a request to all participating nodes asking if they are ready to commit the transaction. Each node performs all necessary checks and writes the transaction changes to a temporary log but does not finalize the commit yet. They then respond with either a “yes” (ready to commit) or “no” (abort). Only if all nodes reply “yes” does the coordinator proceed to the second stage,
· Second the commit phase, where it instructs all nodes to finalize and permanently commit the changes. If any node fails to commit, the coordinator tells all nodes to roll back the transaction or some manual intervention might be needed.
While 2PC guarantees atomicity across nodes, it has some significant drawbacks. Because all nodes must wait for each other’s responses, the protocol can become slow under heavy load, as network communication and disk writes add latency. Moreover, 2PC is a blocking protocol, if any node or the coordinator crashes after the prepare phase but before sending the commit or rollback message, the other nodes remain locked in a waiting state, unable to proceed or release resources. This can cause the entire system to stall unless complex recovery mechanisms or timeouts are implemented, which adds further complexity to distributed transaction management.
Concurrency and Locking: When a database is distributed across multiple nodes, managing concurrency, the ability for multiple transactions to run simultaneously without interfering with each other becomes significantly more challenging. Each node must enforce locks on the data it manages to ensure transactions don’t overwrite or read inconsistent data. As the number of nodes increases, so does the total number of locks held across the system. This higher volume of locks raises the chances of deadlocks. Additionally, contention increases as multiple transactions compete for the same resources, causing some to wait longer before they can proceed. This waiting leads to blocking, where transactions are forced to pause until the necessary locks are released. Together, these issues reduce the system’s overall throughput (the number of transactions completed in a given time) and add complexity to concurrency control mechanisms. Distributed systems must implement efficient algorithms to detect and resolve deadlocks, manage lock timeouts, and optimize resource access, making concurrency control a major challenge as the system scales horizontally.
Consistency Models: Maintaining strong consistency across distributed nodes means ensuring that every node always sees the same up-to-date data at any given time. Achieving this requires constant synchronization between all the nodes so that updates are immediately reflected everywhere. However, in a distributed system, network delays, failures, or partitions, where nodes become temporarily unreachable make perfect synchronization difficult.
According to the CAP theorem, a distributed system can only guarantee two out of following three properties at the same time.
· Consistency,
· Availability, and
· Partition tolerance.
Prioritizing strong consistency often means sacrificing availability because the system must wait for all nodes to synchronize before responding to requests. Alternatively, if the system remains available during network partitions, it may have to allow some nodes to operate on stale data, sacrificing strict consistency. This fundamental trade-off means that maintaining strong consistency in distributed systems is complex and often involves balancing synchronization overhead, response times, and fault tolerance.
In summary, we need an effective and robust distributed transaction manager (TM), which is a critical component for ensuring reliable coordination of transactions that span multiple nodes or services in a distributed system. The TM’s primary responsibility is to orchestrate the entire lifecycle of a distributed transaction, making sure it adheres to ACID properties across all participants. To do this, the TM communicates with all involved resource managers (such as different databases or microservices), coordinating their actions to prepare, commit, or roll back changes in a way that maintains global consistency.
The TM must also handle failure recovery gracefully. This involves detecting and resolving partial failures where some nodes may have committed while others failed, as well as managing situations where the TM or participants crash during the transaction process. It typically uses durable logs to record transaction states and decisions, enabling it to resume or roll back incomplete transactions after a failure.
In addition, the TM often implements standard protocols like Two-Phase Commit (2PC) or more advanced consensus algorithms to ensure reliable communication and coordination. It abstracts the complexity of distributed coordination away from individual resource managers, providing a unified interface that guarantees transactional integrity, even in the case of network delays, node failures, and concurrent access conflicts.
System Design Questions
No comments:
Post a Comment