A distributed system fails in the gaps between its parts, so I design for the part that goes quiet

Most failures in a distributed system happen between its parts rather than inside them, so I design for what each part does when another one goes quiet, answers late or reaches for the same state.

Most of what goes wrong in a distributed system happens between its parts, not inside them. A worker stops answering without crashing. An answer arrives after the question has changed. Two workers reach for the same piece of state at once. Nodes that do not trust each other disagree about what happened. None of this shows up when each part is tested on its own.

So I start from what each part can know about the others, and from what it does when it cannot know. Every claim on shared work gets a way to expire, every answer a way to be recognised as stale, and every conflict a place where it is predicted rather than discovered. A liveness check watches the work a part produces, not a side effect that can stall for another reason. The articles below each take one case where the gap between parts, not the parts themselves, decided whether the system stayed correct.

No article on this topic is published yet.