A practical explanation of message boundaries, partial failure, ordering, consistency, retries, and the operational cost behind distributed software.
A second computer does more than add capacity. It creates a new kind of uncertainty: the first computer can send a request, receive no reply, and still be unable to tell whether the second computer completed the work.
A distributed system is a set of autonomous computing components that coordinate through messages to produce a shared service or result. The components do not share perfectly reliable communication, one clock, or one failure state. That is why a distributed service can be faster, larger, and more resilient than one machine—and also why it can be harder to understand.
A definition you can test at a system boundary
In the usual software-architecture sense, a distributed system spans multiple networked computers. Leslie Lamport’s classic paper uses a broader and more useful boundary: distinct processes exchange messages, and transmission delay matters compared with the events inside a process. The paper also shows why independent events do not always have one obvious global order.
Use this four-question boundary test instead of looking for a “cloud” label:
- Do two or more components run independently? They may start, stop, or make progress at different times.
- Do they coordinate by sending messages? An HTTP call, database query, queue event, or replication record crosses a boundary.
- Can one participant fail or become unreachable while another continues? This creates a partial failure rather than one clean system-wide stop.
- Must the participants agree on a useful result? They may need to agree on an order, balance, owner, lock, inventory count, or latest version.
If a user-visible outcome crosses a message boundary and the participants can observe different state or failure, reason about the job as a distributed operation. A browser calling an application server and database qualifies. So do a content delivery network, a multiplayer game, a replicated database, and a batch-processing cluster. One function calling another in the same process usually does not, even though local concurrent programs can share some ordering problems.
“Autonomous” does not mean the components belong to different companies. It means each component has its own execution and failure timing. Ten services on one physical host may be logically distributed but still share one hardware failure domain; two replicas in separate regions add physical separation as well.
Follow one checkout to see the real problem
Imagine a hypothetical store where one checkout request touches an API, inventory service, payment service, order database, and receipt worker. The customer sees one button. The system sees a sequence of messages and local state changes.
| Participant | Its local fact | What it cannot know by itself |
|---|---|---|
| Checkout API | It accepted request order-417. |
Whether payment completed after the reply timeout. |
| Inventory | Two units are reserved for order-417. |
Whether the customer was charged. |
| Payment service | The $80 charge committed. | Whether its success reply reached the API. |
| Order database | No completed order exists yet. | Whether an earlier service has state that requires recovery. |
| Receipt worker | No receipt event arrived. | Whether it should wait, retry, or compensate. |
The API timed out, but “timeout” describes what the caller observed—not what happened at the payment service. Amazon’s Builders’ Library discussion of distributed-system challenges treats this unknown network outcome as a core operational problem: a request may have failed, succeeded, or arrived without being processed.
A blind retry might charge the customer twice. Abandoning the request might leave money captured without an order. A defensible design gives the operation an idempotency key, stores each durable transition, makes duplicate delivery safe, and runs reconciliation for incomplete orders. If the workflow cannot complete, it needs an explicit compensating action such as releasing inventory or reversing a payment—not a hope that every component rolls back at once.
This failure ledger is the central mental model: each participant knows its own durable state, messages carry claims about other state, and recovery must close the gaps.
Five constraints shape every distributed design
Messages take time
A local memory read and a cross-region request do not have the same cost or variance. Each network hop adds serialization, queues, transport, processing, and a return path. A chain of individually acceptable calls can produce a slow user action. ToolMerit’s guide to latency separates round-trip delay from server work and transfer time, which is useful when a distributed request feels slow.
Failures are partial and sometimes ambiguous
A node can crash while its peers continue. A network can separate healthy nodes. A request or reply can be delayed, duplicated, or lost. The observer often cannot distinguish a slow dependency from a failed one, so timeouts are policy decisions rather than proof of remote failure.
There is no free global “now”
Machine clocks drift, messages arrive out of order, and concurrent events may have no causal relationship. Lamport’s work on logical clocks and the happened-before relation explains why event order is only partial unless a system adds a coordination rule. A database sequence, version vector, leader, or consensus protocol can impose the order required for a particular job, but coordination costs time and can reduce availability during faults.
Copies create consistency choices
Replication improves read capacity and can preserve service when one copy fails. It also creates a question: when is a write visible at every copy? Some jobs require the latest committed value before responding. Others can safely show an older value briefly. “Eventual consistency” is not a blanket quality setting; it is a promise that should name which data may lag, for how long operationally, and what the user can do while views disagree.
Recovery traffic can amplify a small fault
When many clients retry the same overloaded service immediately, recovery traffic can deepen the overload. A failed replica can also transfer load to its healthy peers. Timeouts, bounded retries, exponential backoff, jitter, circuit breaking, load shedding, and isolation are ways to control that feedback—not decorations added after launch.
CAP is a partition-time rule, not a product score
The CAP theorem is often reduced to “pick two of three,” which hides the useful condition. Gilbert and Lynch’s result concerns what a distributed service can guarantee when the network is partitioned: it cannot simultaneously provide a strongly consistent view and a successful response for every request on both sides. Their later perspective on CAP places that tradeoff in the wider context of distributed computing.
In practice, choose per operation. A bank may reject or delay a balance-changing write rather than accept conflicting values. A product catalog may continue serving a slightly stale description. The decision is not “is this whole system CP or AP?” but “during this partition, which operations must preserve one truth, and which can remain useful with bounded staleness?”
The recurring parts of a distributed system
Distributed systems take many forms, but the same responsibilities recur:
| Responsibility | Common mechanism | New question it creates |
|---|---|---|
| Find a healthy worker | Service discovery, routing, load balancing | How quickly is unhealthy membership removed? |
| Keep work available | Replication and failover | Which copy is authoritative after a split? |
| Spread a large dataset | Partitioning or sharding | What happens when one shard becomes hot or unavailable? |
| Separate producers from consumers | Queues, logs, and event streams | Can delivery repeat, and where is progress recorded? |
| Choose one owner or order | Leases, leader election, consensus | How is a stale owner prevented from writing? |
| Explain one user request | Correlated logs, metrics, and distributed traces | Does context survive every process and queue boundary? |
The OpenTelemetry specification overview defines a distributed trace as events that cross process, network, and security boundaries. That is why a single request ID or trace context matters: without correlation, five locally correct logs can still fail to explain one broken checkout.
Distributed is not the same as cloud or microservices
| Term | What it describes | Relationship to distribution |
|---|---|---|
| Distributed system | Independent components coordinate through messages. | The underlying computing property. |
| Microservices | An application is organized as small, independently deployable services around business capabilities. | Usually distributed, but a design choice rather than a synonym. |
| Cloud computing | Computing resources are delivered and managed as services. | Can host one application instance or a large distributed platform. |
| Parallel computing | Multiple workers execute parts of a computation at the same time. | May be distributed across machines or local to one machine. |
| Decentralized system | Control or authority is intentionally spread among participants. | Distributed physically or logically, with an added governance property. |
Microsoft’s microservices architecture guidance illustrates the distinction. Independent deployment and scaling can be valuable, but the whole system gains service-discovery, data-consistency, transaction, network-latency, testing, and observability work. Splitting an application into more services does not automatically make it more reliable.
Use distribution only when its job pays the coordination cost
Distribution is justified when a real requirement exceeds a simpler system: one machine cannot handle the data or traffic; the service must survive a machine or location failure; users need computation near them; one workload must scale independently; or organizational boundaries require separately operated components.
Prefer a single deployable application and one authoritative database when they comfortably meet the load, availability, geography, and team-boundary requirements. A modular monolith can preserve code boundaries without turning every function call into a fallible network operation.
ToolMerit’s rule is simple: do not count nodes; count failure semantics you are prepared to own. Every new remote boundary needs an answer for deadline, retry safety, duplicate delivery, source of truth, consistency, access control, observability, and recovery. If the team cannot name those answers, the architecture has spent complexity it has not budgeted to operate.
Classify a remote result before deciding to retry
For an unfamiliar system, draw each component as a box containing only the state it owns. Draw every network call or event as an arrow. Then take one user action and annotate each arrow with four possibilities: delayed, duplicated, lost before processing, and processed with the reply lost. Mark the authoritative state and the recovery owner.
Finally, classify each remote result as known success, known failure, or unknown. A known failure can follow an explicit repair path. An unknown outcome must first be queried, deduplicated, or reconciled; merely increasing a timeout does not make the ambiguity disappear.
Apply that exercise to the most consequential action in the system—charge, publish, delete, reserve, or grant access. If you can show what every participant knows after each message and how the system closes an incomplete state, you understand the distributed system more accurately than a diagram containing a hundred unlabeled services.
