Distributed Systems Engineering Principles#

Introduction: Starting from three “luxuries”

When writing single-machine programs, there are three things you take for granted, never realizing they are luxuries: function calls always succeed, there is a single unified “now” for the whole world, and every operation either succeeds or fails. The moment you cross two machines, all three vanish—the network drops, reorders, duplicates, and even partitions messages; every machine’s clock differs, and “simultaneity” becomes an illusion; and most critically, the outcome of an operation is no longer just “success” or “failure,” but a third state: unknown.

This is the fundamental difficulty of distributed systems. It is not that they become “more complex” because there are more machines and greater scale, but that the basic assumptions of the system undergo a qualitative change. Understanding distributed systems engineering principles is essentially about understanding how these assumptions change, and how to build reliable systems on top of them.

Partial Failure: The Root of All Evil#

Determinism on a Single Machine vs. Non-Determinism in a Distributed System#

In a single-machine environment, when the hardware is working properly, the same operation always produces the same result—this is determinism. A program either runs or crashes; failure is atomic and observable. In a distributed system, however, some parts of the system may fail in unpredictable ways while other parts continue to work normally—this is partial failure, and it is non-deterministic.

The difficulty of partial failure lies not in its being “more severe,” but in its being “undecidable.” You send a request to another machine and hear nothing back for 2 seconds—did it not receive it? Did it receive it but is still computing? Did it finish but the response was lost? Or has the whole machine gone down? You cannot distinguish “it is dead” from “it is merely slow.” This gray failure is the core contradiction that reliability design for the entire distributed system must face.

The Nature of Timeout: A Guess#

Faced with gray failure, your only weapon is the timeout: wait long enough and if there is still no response, treat it as a failure. But a timeout is essentially a guess.

Guess too short, and you will misjudge a merely slow node as dead, triggering unnecessary retries and primary-secondary failovers, making matters worse; guess too long, and failure recovery becomes sluggish and the blast radius grows. Theoretically, a reasonable timeout value is 2d + r (where d is the maximum network delay and r is the node processing time), but these values can hardly be defined accurately in practice, so they usually require continuous measurement and dynamic adjustment according to jitter.

A Mental Flip in Architecture: In the single-machine era, you write code assuming calls will succeed; in the distributed era, you must assume that every cross-network call may fail, time out, or even be delivered repeatedly. Then ask yourself: under such conditions, what happens to my system? This mental flip is the dividing line between beginner and advanced.

Unreliable Networks and Unreliable Clocks#

In a distributed system, nodes can only communicate through the network, and the Internet and internal data center networks are usually asynchronous packet networks: one node can send a message to another node, but the network does not guarantee when the message arrives, or even whether it arrives at all.

Accompanying this is the clock problem. Clocks on different machines cannot be perfectly synchronized, and a machine’s clock may jump forward or backward, with no guarantee of monotonic increase. Even Google Spanner, using GPS plus atomic clocks, can only keep the clock error between different machines within a millisecond-level range. This means you cannot rely on physical clocks to determine the order of events—precisely why logical clocks (such as Lamport clocks and hybrid clocks) and causality tracking are necessary.

Consistency: Not a Switch, but a Spectrum#

From CAP to PACELC#

The CAP theorem states that a distributed system cannot simultaneously satisfy Consistency, Availability, and Partition tolerance—at most two of the three. Since network partitions are unavoidable in practice, partition tolerance is effectively mandatory, and the real trade-off occurs between consistency and availability.

But CAP has an important limitation: it only describes the trade-off when a partition occurs (a relatively rare event), and ignores the other kind of trade-off the system faces during normal operation. The PACELC theorem extends it:

• When a partition occurs (P), the system must trade off between availability and consistency (A and C)
• When there is no partition (E, i.e., Else), the system still must choose between latency and consistency (L and C)

For example, if a distributed e-commerce system synchronizes order information to all nodes to ensure data consistency, write latency increases significantly. PACELC reminds us: even in the “peacetime” when the network is fully normal, the cost of consistency is always present—it simply manifests as latency.

flowchart TB
    CAP{"CAP"}
    CAP --> C["一致性"]
    CAP --> A["可用性"]
    CAP --> P["分区容错性"]

    P -->|实践中必选| PACELC{"PACELC"}
    PACELC -->|分区时 P| PC["权衡 A 与 C"]
    PACELC -->|否则 E| EL["权衡 L 与 C"]

The Spectrum of Consistency#

Consistency is not an either-or switch, but a spectrum. From strongest to weakest, it roughly includes:

Linearizability is the strongest consistency model. The distributed consensus algorithms that implement linearizability are mainly Paxos and Raft.

The choice of consistency model is essentially an engineering decision: stronger consistency means higher coordination cost, higher latency, and lower availability. The key is not to “pursue the strongest,” but to understand the business’s tolerance for data consistency and strike the optimal balance under constraints.

Order and Causality: A Deep Theme of Distributed Systems#

In distributed systems, “order” is a fundamental theme that recurs repeatedly. The definition of linearizability implies that “all operations form a definite execution order.”

Order matters because it preserves causality. In consistent prefix reads, if a reader sees the answer before the question, this violates causal order—because only after seeing the question could one possibly give the answer. In concurrent write detection, for two operations A and B, there are only three possibilities:

1. A happened before B
2. B happened before A
3. A and B are concurrent

If A happened before B, then B may know about A and build upon it; if A and B are concurrent, then we can conclude they know nothing of each other.

This leads to a profound conclusion: in a distributed system, a node within the network cannot directly be certain of anything; it can only infer from the information it obtains from the network (receiving no reply is also a kind of information), and can only determine state by exchanging information with other nodes. Truth is not “observed”—it is “defined” by the majority.

Logical Clocks and Vector Clocks#

Since physical clocks are unreliable, how do we determine the order of events? Logical clocks provide the answer. The Lamport clock assigns a monotonically increasing counter to each event: each time a node experiences an event, the counter increments by one; when sending a message, it attaches the current count; upon receipt, the receiver takes max(local count, message count) + 1. This yields a total order, but it is only a linear extension of a partial order—if a → b, then C(a) < C(b), but not conversely: C(a) < C(b) does not imply that a causally precedes b.

Vector clocks solve this defect. Each node maintains a vector recording the event counts it knows for itself and other nodes. By comparing two vectors, one can precisely determine three relationships: a precedes b, b precedes a, or the two are concurrent. Inability to order concurrent events means conflict exists—this is precisely the basis for detecting write conflicts in multi-leader replication. The Hybrid Logical Clock (HLC) combines physical and logical clocks, staying close to real time while preserving causality, and is adopted by systems such as CockroachDB.

Replication, Sharding, and Consensus#

Replication Strategies: Single-Leader, Multi-Leader, and Leaderless#

Replication means copying the same data across multiple machines connected by a network. Its benefits are obvious:

• Improved availability (failure of some nodes does not affect the whole)
• Increased read throughput (multiple replicas can share read requests)
• Reduced latency (read from a nearby replica)

But replication introduces a core contradiction: how do you keep all replicas in a consistent state at all times? Imagine a simple key-value store: a client writes value “X” to node A, and due to network delay, the write is not synchronized to node B in time. Another client then reads the same key from node B and gets a stale value, causing data inconsistency. More seriously, if node A crashes before synchronization completes, the data may be lost permanently.

Replication itself is not a single scheme but a class of schemes. Three typical strategies each have their trade-offs:

flowchart LR
    subgraph SL["单主复制"]
        C1["客户端"] --> L1["主节点"]
        L1 --> F1["从节点 A"]
        L1 --> F2["从节点 B"]
    end

    subgraph ML["多主复制"]
        C2["客户端"] --> M1["主节点 A"]
        C3["客户端"] --> M2["主节点 B"]
        M1 <--> M2
    end

    subgraph LL["无主复制"]
        C4["客户端"] --> R1["副本 1"]
        C4 --> R2["副本 2"]
        C4 --> R3["副本 3"]
    end

The degree of synchronization in replication is another independent trade-off line. Synchronous replication guarantees a successful return only after followers confirm, but latency is dragged down by the slowest replica; asynchronous replication has low latency, but unsynchronized writes may be lost if the leader fails; semi-synchronous replication (such as MySQL semi-sync) strikes a balance: at least one follower confirms, combining low latency with a certain degree of data safety.

flowchart TB
    subgraph Sync["同步复制"]
        S1["写入"] --> S2["等待所有从节点确认"]
        S2 --> S3["返回成功"]
    end

    subgraph Async["异步复制"]
        A1["写入"] --> A2["立即返回成功"]
        A2 --> A3["后台同步"]
    end

    subgraph Semi["半同步复制"]
        M1["写入"] --> M2["等待至少一个从节点确认"]
        M2 --> M3["返回成功"]
    end

The Quorum Mechanism: The Mathematics of Majority#

The Quorum mechanism was proposed by David K. Gifford in 1979. Its core idea draws on the pigeonhole principle: if there are N replicas, a write must successfully update at least W replicas to be acknowledged, and a read must read from at least R replicas. As long as W + R > N holds, the write set and the read set must intersect—the read will encounter at least one replica containing the latest data.

The engineering significance of this constraint is that it turns the consistency guarantee into a configurable parameter choice. Once N is fixed, different combinations of W and R produce different consistency-performance-availability trade-offs:

The essence of Quorum is not “guaranteeing strong consistency,” but turning “whether reads and writes overlap” into a computable constraint. When W+R>N, the system behaves close to strong consistency (at least the read encounters the latest write); when W+R≤N, it degrades to eventual consistency. In Dynamo-style systems, clients can choose different W and R per operation—high R for consistency-sensitive reads, low W for latency-sensitive writes.

Sharding: The Basic Means of Horizontal Scaling#

When data volume or request volume exceeds the capacity of a single machine, sharding (also called partitioning) becomes the basic means. Sharding divides data according to some rule (such as key range or hash value) across different nodes, each responsible for a portion of the data.

The core challenge of sharding is not “how to divide,” but “how to coordinate after dividing.” When an operation spans multiple shards, a distributed transaction problem arises.

Consistent hashing is one of the most important algorithms in sharding and load balancing. It maps both servers and requests onto a hash ring, routing requests to the nearest server clockwise. When nodes are added or removed, only a portion of the requests of adjacent nodes need to be migrated, rather than remapping everything—this solves the problem that naive modulo hashing almost completely fails when nodes change.

Distributed Transactions: Atomicity Across Boundaries#

The ACID of single-machine transactions is guaranteed by the database engine. Once an operation spans multiple databases or services, atomicity requires an additional coordination mechanism. Distributed transaction schemes fall roughly into two categories: rigid (pursuing strong consistency) and flexible (accepting eventual consistency).

Two-Phase Commit (2PC) is the most classic rigid scheme. The coordinator first sends “prepare to commit” to all participants; participants execute the operation but do not commit, lock resources, and write logs, then return “ready” or “abort”; once all are ready, the coordinator sends “commit,” otherwise it sends “rollback.” The cost of 2PC is synchronous blocking: locks acquired during the Prepare phase are held until the second phase completes, and the relevant data is invisible to other transactions throughout. More fatally, there is the coordinator single point of failure: if the coordinator crashes in the window “after phase one completes but before phase two is sent,” all participants hang indefinitely. The industry’s response is not 3PC (an extra round trip yields limited benefit), but rather using a consensus protocol to make the coordinator highly available.

sequenceDiagram
    participant C as 协调者
    participant P1 as 参与者 1
    participant P2 as 参与者 2

    C->>P1: Prepare
    C->>P2: Prepare
    P1-->>C: Ready
    P2-->>C: Ready
    C->>P1: Commit
    C->>P2: Commit
    Note over C,P2: 若协调者在 Commit 前崩溃，<br/>参与者将无限期悬挂

TCC (Try-Confirm-Cancel) moves the logic of 2PC from the database layer to the business layer. Each participant implements three interfaces: Try performs resource reservation (freezing inventory, pre-occupying credit), Confirm performs the final commit, and Cancel performs release and rollback. TCC’s advantage is finer lock granularity and smaller concurrency impact, because “reservation” is at the business-semantics level, not a database row lock. The cost is extremely high business intrusiveness: each operation requires three sets of interfaces, and one must also handle three engineering problems—idempotence, empty rollback, and dangling transactions.

flowchart LR
    T["Try：预留资源"] --> C["Confirm：提交"]
    T --> X["Cancel：释放"]
    C -.->|若后续步骤失败| Comp["补偿动作"]

Saga targets long-running processes. It splits a global transaction into a series of local transactions, each paired with a compensating action; the forward flow executes in sequence, and if any step fails, the compensating actions of the already-successful steps are executed in reverse. Saga’s core idea is to give up atomicity and trade compensation for availability—it does not lock resources, and each local transaction commits independently, making it suitable for cross-system processes lasting from seconds to minutes. The cost is that compensation logic must be implemented by the business, and compensation itself may fail, requiring manual fallback.

flowchart LR
    S1["步骤 1"] --> S2["步骤 2"]
    S2 --> S3["步骤 3"]
    S3 --> S4["步骤 4"]
    S4 -.->|失败| C3["补偿步骤 3"]
    C3 --> C2["补偿步骤 2"]
    C2 --> C1["补偿步骤 1"]

Local message table / transactional messaging is an engineering solution for eventual consistency: the local business operation and the “message to be sent” are written in the same local transaction, then the message is delivered asynchronously, and the downstream confirms after consumption. It does not pursue atomicity, only the guarantee that the message will eventually be delivered.

flowchart LR
    B["业务操作"] --> T["同一个本地事务"]
    M["消息记录"] --> T
    T --> P["异步投递"]
    P --> D["下游消费"]
    D --> A["确认"]

Consensus: The Most Important Abstraction in Distributed Systems#

Consensus is about getting all nodes to agree on something. It is one of the most important abstractions in distributed systems and the cornerstone of building strongly consistent systems.

Raft is the consensus algorithm most widely adopted in current engineering practice. It was proposed by Diego Ongaro and John Ousterhout in 2013, with the design goal of providing an “easy-to-understand” consensus algorithm. Raft’s core idea is to decompose the consensus problem into three relatively independent subproblems:

1. Leader election
2. Log replication
3. Safety

and to adopt a strong leader model to simplify the entire process.

Raft is based on the replicated log state machine model. Its core principle is: if two identical deterministic processes start from the same state and receive the same inputs in the same order, they will produce the same outputs and end in the same state. Therefore, Raft’s most important task is to ensure that the logs of all replicas (time-ordered, append-only sequences of records) remain consistent. The leader organizes client requests into log entries, and only after a majority of nodes have acknowledged an entry does the leader commit it, with all replicas executing requests in the order of the log.

flowchart LR
    C["客户端"] --> L["领导者"]
    L --> Log["追加日志条目"]
    Log --> F1["跟随者 1"]
    Log --> F2["跟随者 2"]
    Log --> F3["跟随者 3"]
    F1 --> ACK["多数派确认"]
    F2 --> ACK
    ACK --> Commit["提交条目"]
    Commit --> Apply["应用到状态机"]

Raft is a CP system—when facing a network partition, it chooses to guarantee consistency and may sacrifice some availability. When the cluster splits into two subsets that cannot communicate, only the subset containing the majority of nodes can continue to provide service; the other subset stops accepting writes, avoiding split-brain.

Raft has been widely applied in numerous distributed systems:

• Key-value stores such as Etcd and Consul use it to guarantee strong consistency and high availability
• Distributed databases such as CockroachDB use it as the core of multi-shard consistency
• In the field of distributed lock services, Google Chubby and Azure Service Bus also rely on similar mechanisms

The Boundary of Consensus: When You Don’t Need Consensus#

Consensus is one of the most powerful tools in distributed systems, but it is also the most expensive tool. Understanding “when you don’t need consensus” demonstrates judgment better than “how to implement consensus.”

Not all systems need consensus. Leaderless and multi-leader replication systems typically do not use global consensus; they accept the existence of conflicts—the result of different leaders failing to reach consensus is that data develops branching and merging version histories. This is not a problem: perhaps we simply need to accept the fact that there is no linearizability, and learn to work better with data that has branching and merging version histories.

Tools like ZooKeeper provide applications with “outsourced” consensus, failure detection, and membership services. If you find that the problem you want to solve can be reduced to consensus, and you want it to be fault-tolerant, using ZooKeeper or etcd is wise. But if you merely need a distributed configuration store, or an eventually consistent cache, consensus is using a sledgehammer to crack a nut.

The FLP impossibility result theoretically delimits the boundary of consensus: in an asynchronous system, even with only one faulty process, there is no deterministic algorithm that can guarantee consensus. This tells us not to waste time trying to design a consensus algorithm for arbitrary scenarios in an asynchronous system. The engineering response is to bypass it: convert the asynchronous system into a synchronous one (by assuming timeouts), or use randomized algorithms. The lesson of FLP is: science tells you what is impossible, while engineering tells you that at some cost you can make it feasible.

Engineering Practice: Fault-Tolerance Patterns and Design Patterns#

The Eight Fallacies of Distributed Systems#

The eight fallacies of distributed systems expose the root of all distributed failures:

1. The network is reliable
2. Latency is zero
3. Bandwidth is infinite
4. The topology does not change
5. Transport is secure
6. The network is homogeneous
7. There is one administrator
8. Transport cost is zero

These assumptions are almost always true in single-machine programming, yet almost always false in a distributed environment. In reality, network jitter, machine crashes, dependency timeouts, resource exhaustion, and data inconsistency are the norm rather than the exception in distributed systems.

Failure Detection#

Failure detection is the first gate of the fault-tolerance system. Its core challenge is finding the optimal balance between false negatives (a failure goes undetected and traffic keeps hitting the failed node) and false positives (a healthy node is misjudged as failed, triggering unnecessary traffic removal and failover).

A common cognitive误区 is confusing TCP Keepalive with application-level heartbeats. TCP Keepalive is a transport-layer keep-alive mechanism that by default sends a probe packet every two hours; it can only verify whether the network link of the TCP connection is unobstructed, and cannot perceive application-layer state. For example, when an application process deadlocks, the TCP connection remains in the ESTABLISHED state, but the business can no longer process requests at all—at this point TCP Keepalive will judge the connection as healthy, causing a serious false negative. Application-level heartbeats, by contrast, can verify the health of the application process, business thread pools, and core dependencies, and are the core foundation of failure detection in distributed systems.

In terms of detection mechanisms, fixed-timeout detection cannot adapt to dynamic network environments. The industry-standard solution for adaptive failure detection is the Phi Accrual Failure Detector, which has been deployed at scale in mainstream distributed components such as Akka and Cassandra. Its core logic is to compute a phi value based on the latency distribution of historical heartbeats—representing the confidence that a node has failed; the higher the phi value, the greater the probability of failure.

Fault-Tolerance Patterns: Idempotence, Retry, Circuit Breaking, Degradation#

Idempotent design is the most fundamental safeguard in the fault-tolerance system. In a distributed environment, a timed-out request is like “Schrödinger’s cat”—it may have succeeded or failed. For very important requests, it is generally designed to be idempotent so that it can be safely retried on timeout. In Martin Fowler’s distributed systems pattern catalog, the “Idempotent Receiver” pattern is precisely the systematic expression of this idea: identifying a client’s request through a unique identifier, thereby ignoring duplicate requests when the client retries.

sequenceDiagram
    participant C as 客户端
    participant S as 幂等接收器
    C->>S: 请求（id = 123）
    S-->>C: 成功
    C->>S: 重试（id = 123）
    Note over S: 识别重复 id
    S-->>C: 成功（不重复副作用）

The design of a retry strategy must be coordinated with idempotence:

• Retries should be strictly limited in number (usually no more than 3)
• Retries should be configured only for idempotent operations (GET, PUT, DELETE, etc.)
• Each retry should have timeout control to avoid long blocking
• More sophisticated schemes use exponential backoff with jitter, combined with a retry budget mechanism—automatically stopping retries when the system as a whole is unhealthy, preventing further load amplification at the moment when capacity is most scarce

flowchart LR
    R["请求失败"] --> B["指数退避 + 抖动"]
    B --> Q{"预算是否充足？"}
    Q -->|是| Retry["重试"]
    Q -->|否| Stop["停止重试"]
    Retry --> R

Circuit breaking and degradation are key to preventing failure propagation:

• Circuit breaker: automatically cuts off traffic when consecutive failures of a downstream service are detected, giving the downstream time to recover
• Degradation: when resources are insufficient or dependencies are unavailable, sacrifices non-core functionality to keep the core path available

The truly stable production practice is the combined coordination of “reasonable timeouts + limited retries + idempotent fallback + circuit breaking and degradation,” rather than relying on any single measure.

Distributed Systems Pattern Catalog#

In Patterns of Distributed Systems, published on Martin Fowler’s website and in book form in 2023, Unmesh Joshi systematically organizes 30 distributed systems patterns, covering everything from clock handling to data partitioning. These include:

The common characteristic of these patterns is that they are all engineered responses to the inherent difficulties of distributed systems, rather than encapsulations of some specific technology. Understanding these patterns has more long-term value than learning to use a particular framework.

Observability: The Eyes of a Distributed System#

In single-machine programs, debugging relies on breakpoints, logs, and stack traces. In a distributed system, a request crosses a dozen services, and a failure may occur at any link—observability is not “nice to have,” but a prerequisite for being able to operate.

Observability has three pillars:

• Metrics: numerical measurements and counters, answering “what is happening.” CPU utilization, request rate, error rate, latency percentiles—their advantage lies in aggregation and alerting: you can set an alert for “error rate exceeds 1%” without reading any logs.
• Logs: timestamped records of events, answering “why is this happening.” Structured logs (JSON format) are easier to query and correlate than unstructured text. But the problem with logs is noise and silos: most logs are irrelevant most of the time, while key logs are scattered across different storage systems of different services.
• Distributed tracing: the one that best embodies the characteristics of distributed systems. It traces the complete path of a request across all services, answering “where did it happen.” The principle is context propagation: the caller injects a trace ID and span ID into the request header (in the W3C TraceContext standard format traceparent: 00-{trace-id}-{span-id}-{flags}), and the callee extracts them and continues propagation as a child span. In this way, operations originally isolated in each service are linked into a complete call chain.

flowchart LR
    subgraph Pillars["三大支柱"]
        M["指标<br/>正在发生什么"]
        L["日志<br/>为什么会这样"]
        T["追踪<br/>发生在哪里"]
    end

sequenceDiagram
    participant U as 用户
    participant G as 网关
    participant A as 服务 A
    participant B as 服务 B
    U->>G: 请求（trace-id, span-id）
    G->>A: 传播 trace-id, 新建 span
    A->>B: 传播 trace-id, 新建子 span
    B-->>A: 返回
    A-->>G: 返回
    G-->>U: 返回
    Note over U,B: 所有 span 共享同一 trace-id，可重建完整调用链

Distributed tracing is especially critical for AI applications. In a multi-agent system, an orchestrator agent calls a product search agent, a payment agent, and a logistics agent. Without explicit trace context propagation, each agent creates a new trace ID and correlation is lost. With correct propagation, the spans of all agents share the same trace ID, allowing the complete interaction flow to be reconstructed. Debugging agent orchestration is, in essence, a distributed tracing problem.

flowchart LR
    O["编排 Agent"] --> S["搜索 Agent"]
    O --> P["支付 Agent"]
    O --> L["物流 Agent"]
    S -.->|共享 trace-id| T["统一追踪"]
    P -.->|共享 trace-id| T
    L -.->|共享 trace-id| T

Load Balancing and Service Discovery: The Dispatch Hub for Requests#

Load balancing is the “dispatch hub”—all external requests must pass through this layer before reaching the node that actually handles them. Its core challenge is balancing four conflicting goals: evenness (distributing requests as evenly as possible across servers), affinity (routing the same user’s requests to the same server as much as possible), responsiveness (allocating by capability rather than evenly), and fault tolerance (automatically removing a server when it goes down).

The algorithms progress from simple to complex:

Service discovery is the “upstream” of load balancing: service instances register with a registry (etcd, Consul, Nacos), and the load balancer obtains the list of healthy instances from the registry. Health checks (application-level heartbeats, not TCP Keepalive) are responsible for removing failed instances. Client-side load balancing makes decisions locally at the caller, while server-side load balancing is handled by a dedicated gateway—the former saves a network hop, the latter centralizes policy management.

An Engineering Perspective on Consistency Guarantees#

CAP and the consistency spectrum are theory. What is truly needed in engineering are usable consistency guarantees that the business can perceive.

Read-your-writes is the consistency requirement most commonly perceived by the business: after a user submits a form and refreshes the page, they must see the data they just submitted. Implementation approaches include: always routing the user’s own writes to the leader; recording the timestamp of the user’s most recent write and allowing reads from a follower only after it has caught up to that timestamp; or having the client remember its own write version and validating it on read.

Monotonic reads ensure that users do not “travel back in time”: if the first read saw new data, the second read must not see older data. Implementation approaches include binding the user to a fixed replica, or recording the minimum version the user has read.

Consistent prefix reads preserve causal order: if a reader sees the answer before the question, causality is violated. It requires that if a series of writes occurs in a certain order, the reader must see them in the same order.

These guarantees are weaker than linearizability and stronger than eventual consistency, but they cover the actual needs of the vast majority of business scenarios. The key to choosing a consistency model is not to “pursue the strongest,” but to understand what the business can tolerate and strike the optimal balance under constraints.

Cost and Capacity: The Economics of Distributed Systems#

Every guarantee in a distributed system carries a price tag. Replication means storage cost multiplied by the number of replicas; cross-region replication means bandwidth cost and cross-region latency; consensus means at least one extra round-trip of latency.

When a single machine is enough: when the data volume fits within a single machine’s storage capacity, the QPS fits within a single machine’s processing capacity, and availability requirements do not require multi-replica redundancy, a single-machine solution is simpler, cheaper, and less error-prone. The complexity cost of distributed systems is real—an extra layer of network, an extra set of coordination, an extra class of failure modes.

When eventual consistency is more appropriate than strong consistency: when the business can tolerate brief data inconsistency (likes on social media, product browsing history, feature updates for recommender systems), the coordination cost of strong consistency yields no return. Trading eventual consistency for write availability and latency is the actual choice of most internet systems.

Conclusion: Judgment Matters More Than Implementation#

The engineering principles of distributed systems ultimately revolve around one core question:

When one part of the system does not know the state of another part, how do you make the right decision?

From the undecidability of partial failure, to the trade-off choices along the consistency spectrum; from the majority defining truth in consensus algorithms, to the idempotent design in fault-tolerance patterns that provides fallback for the “unknown” state—all technical solutions answer different facets of this question.

It is worth emphasizing that today, when AI can rapidly generate Raft implementation code, the real difficulty lies not in “how to implement,” but in “whether consensus should be used at all,” “how much inconsistency can be tolerated,” and “at the moment the network splits, do you choose consistency or availability.” The cost of these judgment calls is borne by the business, and AI cannot provide standard answers, because the answer depends on what you are willing to trade for what.

Implementation is becoming cheaper; judgment is becoming more valuable. This is precisely the significance of understanding distributed systems engineering principles.