Distributed KV Store Part 3

Part 2 gave every node a way to compute a key's replica set independently. Nothing used that computation yet — a PUT still only ever touched the local node it landed on. This part makes that computation load-bearing: any node can now accept a write for any key, forward it to that key's actual replicas, and report success once enough of them have it.

The coordinator is a role, not an identity

There is no leader in this system and no fixed router. A client can send PUT /kv/order:777 to any node in the cluster — it doesn't need to know or care which nodes are actually order:777's replicas. Whichever node receives the request becomes the coordinator for that one request. It uses the same Ring.Replicas(key, N) call from Part 2 — the same thing every other node would compute — to find out who the real replicas are, forwards the write to them, and reports back once enough of them have confirmed it.

The same physical node can be coordinator for one key and a plain replica for another, in the same second, depending only on what the ring says about each key. Nothing in this design distinguishes "coordinator nodes" from "replica nodes" as separate categories — every node runs identical code and can play either role for any given request.

What coordinating a write actually means

Coordinating is fan-out plus counting acknowledgements. Concretely, for a PUT or DELETE:

1. Compute the key's replica set from the ring (N nodes).

2. Assign the write a single timestamp, once, before sending anything.

3. Send the write to all N replicas concurrently — including the coordinator's own store, if it happens to be one of the replicas.

4. Wait until W of them have acknowledged.

5. Return success to the client as soon as W is reached, without waiting for the rest.

6. If too many replicas fail to make W reachable, return an error instead.

That's the entire algorithm. There's no leader election, no write-ahead log shared between nodes, no agreement protocol. It's fan-out and a threshold — which is also exactly why it can't offer strong guarantees a consensus protocol would: a write that gets W acks and returns success has, by construction, possibly not reached all N replicas yet. What happens to the ones it didn't reach is a real question, and this part answers it directly rather than glossing over it.

Assigning the timestamp once, not per replica

Part 1 already flagged that a per-node version counter stops meaning anything once more than one node can write the same key — a replica that missed earlier writes could report version 1 for what's actually the fourth write to that key. This part is where that consequence actually arrives, and it changes something concrete: the coordinator generates the write's timestamp exactly once, before contacting any replica, and sends that same timestamp to every one of them. It does not let each replica call time.Now() independently.

If each replica timestamped its own copy, three replicas could end up with three slightly different timestamps for what a client experiences as a single write — and Part 5's read repair, which decides which of several replicas' values is newest by comparing timestamps, would be comparing clock skew, not write order. A single coordinator-assigned timestamp is what makes "this is the same write, everywhere" a fact instead of an approximation.

One direct consequence: there is no longer a canonical per-write version number the client can be told. Part 1's X-Version header, and the 201-vs-200 distinction that depended on it, both assumed a single number could describe a write. That assumption held for one node. It doesn't hold for N independently-counting replicas, so this part drops it rather than keep a header that would be quietly misleading. What the client gets back instead is the one number every replica actually agrees on — the coordinator-assigned timestamp — plus how many of the N replicas acknowledged.

Implementation checkpoint

• The cluster config gains a write_quorum field (W), validated to be between 1 and the replication factor.

• PUT and DELETE on /kv/{key} go through a coordinator instead of writing to the local store directly.

• A new internal endpoint, PUT /internal/kv/{key}, is what the coordinator actually calls on each replica — including itself, through the same code path, not a special-cased local write.

• The coordinator waits for W acks and returns as soon as it has them; it does not wait for stragglers.

• Replica writes that are still in flight when the coordinator returns are not aborted — they run to completion in the background.

• If W can't be reached, the coordinator returns an error, and does not roll back replicas that already succeeded before that determination was made.

• GET is untouched: it still only reads the local node's own store. A node can return 404 for a key that a quorum write just placed on other replicas, if this node isn't one of them. That gap is what Part 4 closes.

Project layout

distributed-kv-store/
  cmd/kvnode/main.go
  internal/store/store.go
  internal/store/store_test.go
  internal/cluster/config.go
  internal/cluster/config_test.go
  internal/hashring/ring.go
  internal/hashring/ring_test.go
  internal/replicate/protocol.go
  internal/coordinator/coordinator.go
  internal/coordinator/coordinator_test.go
  internal/api/handler.go
  internal/api/cluster_handler.go
  internal/api/cluster_handler_test.go
  internal/api/replica_handler.go
  cluster.json

Two additions: replicate, a tiny package with no logic — just the two struct types that describe the wire format a coordinator and a replica agree on — and coordinator, which owns the fan-out and quorum logic. replicate exists on its own specifically to avoid a dependency problem: the client-facing Handler in api needs to call into coordinator (to make writes), and the new ReplicaHandler in api needs the wire-format types to parse incoming replication requests. If those types lived inside coordinator, that would still work — but putting them in a neutral leaf package with zero dependencies is more honest about what they are: a protocol both sides agree on, not internal state either side owns.

Extending the store: one write path for local and remote

Before any of the coordination logic, the store needs a write method that accepts an explicit timestamp instead of computing its own — otherwise "the coordinator assigns one timestamp" is just a claim, not something the code enforces.

// internal/store/store.go (new method)

// ApplyReplicated stores an entry using a timestamp decided by a write
// coordinator, rather than this node's own wall clock. It is used when
// this node is acting as a replica for a write coordinated elsewhere,
// including the case where the coordinator is itself a replica for the
// key and applies to its own store through this same method instead of
// a separate local code path.
//
// Unlike Delete, ApplyReplicated does not require the key to already
// exist before accepting a tombstone. A replica that missed the
// original PUT — because it was down, or the write hasn't reached it
// yet — must still be able to accept a DELETE forward and record a
// tombstone. Refusing it would leave that replica permanently disagreeing
// with the rest of the cluster about whether the key is gone.
func (s *Store) ApplyReplicated(key string, value []byte, timestamp int64, deleted bool) Entry {
	s.mu.Lock()
	defer s.mu.Unlock()

	prev, exists := s.data[key]
	version := uint64(1)
	if exists {
		version = prev.Version + 1
	}

	var v []byte
	if !deleted {
		v = append([]byte(nil), value...)
	}

	entry := Entry{
		Value:     v,
		Version:   version,
		Timestamp: timestamp,
		Deleted:   deleted,
	}
	s.data[key] = entry
	return cloneEntry(entry)
}

Put and Delete from Part 1 are untouched and still correct — they're just no longer what production code calls. From this part on, every write that actually happens in the running system, local or remote, goes through ApplyReplicated. That's a deliberate invariant, not an accident of refactoring: there is exactly one function in the entire codebase that writes an entry into the map.

The wire format

// internal/replicate/protocol.go
// Package replicate defines the wire format for node-to-node write
// replication. It has no logic of its own — just the request and
// response shapes a write coordinator and the replica on the other end
// both agree on. Keeping this in its own package, rather than in the
// coordinator or the HTTP handler, means neither has to import the
// other just to share a struct definition.
package replicate

// Request is what a write coordinator sends to one replica for a
// single key. Value is base64-encoded so arbitrary bytes travel safely
// as JSON. Timestamp is the write's timestamp as the coordinator
// decided it, not the replica's own clock — every replica for a given
// write receives the same Timestamp. Deleted marks a tombstone.
type Request struct {
	Value     string `json:"value"`
	Timestamp int64  `json:"timestamp"`
	Deleted   bool   `json:"deleted"`
}

// Response reports the version the replica stored the entry at. This is
// a per-node counter (see Part 1) and is not meaningful to compare
// across replicas — two replicas can legitimately report different
// version numbers for what is, semantically, the same write.
type Response struct {
	Version uint64 `json:"version"`
}

The replica-side endpoint

// internal/api/replica_handler.go
package api

import (
	"encoding/base64"
	"encoding/json"
	"net/http"

	"distributed-kv-store/internal/replicate"
	"distributed-kv-store/internal/store"
)

// ReplicaHandler serves node-to-node replication requests. It is not
// meant to be called by clients: PUT and DELETE on /kv/{key} go through
// a Coordinator (see the coordinator package), which calls this
// endpoint on each replica in turn, including possibly itself.
type ReplicaHandler struct {
	store *store.Store
}

func NewReplicaHandler(s *store.Store) *ReplicaHandler {
	return &ReplicaHandler{store: s}
}

func (h *ReplicaHandler) Routes(mux *http.ServeMux) {
	mux.HandleFunc("PUT /internal/kv/{key}", h.handleReplicate)
}

func (h *ReplicaHandler) handleReplicate(w http.ResponseWriter, r *http.Request) {
	key := r.PathValue("key")
	if key == "" {
		http.Error(w, "missing key", http.StatusBadRequest)
		return
	}

	var req replicate.Request
	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
		http.Error(w, "invalid replicate request body", http.StatusBadRequest)
		return
	}
	defer r.Body.Close()

	var value []byte
	if !req.Deleted {
		v, err := base64.StdEncoding.DecodeString(req.Value)
		if err != nil {
			http.Error(w, "invalid value encoding", http.StatusBadRequest)
			return
		}
		value = v
	}

	entry := h.store.ApplyReplicated(key, value, req.Timestamp, req.Deleted)

	w.Header().Set("Content-Type", "application/json")
	if err := json.NewEncoder(w).Encode(replicate.Response{Version: entry.Version}); err != nil {
		http.Error(w, "failed to encode response", http.StatusInternalServerError)
	}
}

This handler has no opinion about quorums, replica sets, or who else is involved in the write. It does exactly one thing: take a request that already has everything decided (value, timestamp, tombstone flag), and apply it. All of the coordination logic sits on the other side of this endpoint, in the coordinator.

The coordinator

// internal/coordinator/coordinator.go
// Package coordinator implements the coordinator role for a write:
// given a key and a value, determine the replica set from the hash
// ring, apply the write to those replicas, and report success once W
// of them have acknowledged it. Coordinator is not a fixed identity —
// any node can coordinate a write for any key, whether or not it is
// itself one of that key's replicas.
package coordinator

import (
	"bytes"
	"context"
	"encoding/base64"
	"encoding/json"
	"errors"
	"fmt"
	"net/http"
	"time"

	"distributed-kv-store/internal/cluster"
	"distributed-kv-store/internal/hashring"
	"distributed-kv-store/internal/replicate"
	"distributed-kv-store/internal/store"
)

// ErrQuorumUnreachable is returned when a write cannot collect
// acknowledgements from W replicas, either because too many replicas
// failed or because the ring returned fewer than W replicas to begin
// with.
var ErrQuorumUnreachable = errors.New("write quorum unreachable")

// Result reports the outcome of a successful quorum write. Version is
// deliberately absent here: each replica keeps its own local version
// counter (Part 1), and once more than one replica can hold a key,
// those counters can legitimately disagree — a replica that missed
// earlier writes might report version 1 for what is actually this
// key's fourth write. Timestamp is the one value every replica for this
// write agrees on, because the coordinator assigns it once and sends
// the same value to all of them.
type Result struct {
	Timestamp int64
	Acks      int
	Total     int
}

// Coordinator accepts a write on behalf of a client, computes the
// replica set for the key, and applies the write to W of those
// replicas before returning.
type Coordinator struct {
	selfID  string
	ring    *hashring.Ring
	cfg     cluster.Config
	store   *store.Store
	client  *http.Client
	timeout time.Duration
}

// New builds a Coordinator. selfID identifies which entry in cfg's
// replica set is "this node," so writes to it can go straight to the
// local store instead of over HTTP to itself.
func New(selfID string, ring *hashring.Ring, cfg cluster.Config, s *store.Store) *Coordinator {
	return &Coordinator{
		selfID:  selfID,
		ring:    ring,
		cfg:     cfg,
		store:   s,
		client:  &http.Client{},
		timeout: 2 * time.Second,
	}
}

// Put coordinates a replicated write of value for key.
func (c *Coordinator) Put(ctx context.Context, key string, value []byte) (Result, error) {
	return c.write(ctx, key, value, false)
}

// Delete coordinates a replicated tombstone write for key. It does not
// report whether the key previously existed — that requires reading
// current state across a quorum first, which doesn't exist until
// Part 4. Delete always attempts to write a tombstone to W replicas and
// reports whether that quorum was reached, nothing more.
func (c *Coordinator) Delete(ctx context.Context, key string) (Result, error) {
	return c.write(ctx, key, nil, true)
}

// write fans a value or tombstone out to a key's replica set and blocks
// until W of them have acknowledged it, or until enough have failed
// that W is provably unreachable.
//
// ctx is accepted for symmetry with the rest of the codebase — callers
// naturally have a request context to pass — but it is not threaded
// into the replica fan-out below, and the wait loop does not select on
// ctx.Done(). A caller disconnecting mid-write does not stop this
// function from finishing its quorum decision. Making the wait itself
// cancellable is a reasonable future improvement; it just isn't one
// this part needs to make the coordinator correct.
func (c *Coordinator) write(ctx context.Context, key string, value []byte, deleted bool) (Result, error) {
	replicas := c.ring.Replicas(key, c.cfg.ReplicationFactor)
	w := c.cfg.WriteQuorum

	if len(replicas) < w {
		return Result{}, fmt.Errorf("%w: only %d replicas available for key, need %d",
			ErrQuorumUnreachable, len(replicas), w)
	}

	timestamp := time.Now().UnixNano()

	type ackResult struct {
		err error
	}
	results := make(chan ackResult, len(replicas))

	// Each replica write runs against its own context, detached from
	// ctx, with a fixed per-replica timeout. This function returns as
	// soon as W replicas have acknowledged, without waiting for the
	// rest — but "returned early" does not mean the remaining requests
	// should be aborted. A replica that's merely slower than the other
	// two is still going to succeed if it's given the chance, and there
	// is no reason to throw that write away. Deriving from ctx instead
	// would be actively wrong here: ctx is the incoming HTTP request's
	// context, and net/http cancels that the moment the handler
	// function returns — which is exactly when this function returns
	// after reaching quorum. Any replica write still in flight at that
	// instant would be cancelled by the network layer regardless of
	// how much progress it had made.
	for _, id := range replicas {
		id := id
		go func() {
			bgCtx, cancel := context.WithTimeout(context.Background(), c.timeout)
			defer cancel()
			_, err := c.applyOne(bgCtx, id, key, value, timestamp, deleted)
			results <- ackResult{err: err}
		}()
	}

	acks, failures := 0, 0
	maxFailures := len(replicas) - w

	for i := 0; i < len(replicas); i++ {
		res := <-results
		if res.err == nil {
			acks++
			if acks >= w {
				return Result{Timestamp: timestamp, Acks: acks, Total: len(replicas)}, nil
			}
			continue
		}

		failures++
		if failures > maxFailures {
			return Result{Timestamp: timestamp, Acks: acks, Total: len(replicas)},
				fmt.Errorf("%w: got %d acks (needed %d) out of %d replicas",
					ErrQuorumUnreachable, acks, w, len(replicas))
		}
	}

	// Not reachable: by the time every replica has responded,
	// acks+failures == len(replicas), and the two early-return branches
	// above are complementary, so one of them always fires first. Kept
	// as an explicit error rather than an unreachable panic so that a
	// future change to the accounting above fails loudly here instead
	// of silently returning a zero Result.
	return Result{Timestamp: timestamp, Acks: acks, Total: len(replicas)},
		fmt.Errorf("%w: got %d acks (needed %d) out of %d replicas",
			ErrQuorumUnreachable, acks, w, len(replicas))
}

// applyOne applies one replica's copy of the write, either directly
// against the local store or over HTTP to a remote node. Both branches
// end up calling store.ApplyReplicated with the exact same arguments —
// the local branch just skips the network hop. There is exactly one way
// an entry is ever written, regardless of which node is coordinating.
func (c *Coordinator) applyOne(ctx context.Context, nodeID, key string, value []byte, timestamp int64, deleted bool) (uint64, error) {
	if nodeID == c.selfID {
		entry := c.store.ApplyReplicated(key, value, timestamp, deleted)
		return entry.Version, nil
	}

	addr, ok := c.cfg.AddrOf(nodeID)
	if !ok {
		return 0, fmt.Errorf("no address for node %q", nodeID)
	}

	reqCtx, cancel := context.WithTimeout(ctx, c.timeout)
	defer cancel()

	body, err := json.Marshal(replicate.Request{
		Value:     base64.StdEncoding.EncodeToString(value),
		Timestamp: timestamp,
		Deleted:   deleted,
	})
	if err != nil {
		return 0, fmt.Errorf("encoding replicate request: %w", err)
	}

	url := fmt.Sprintf("http://%s/internal/kv/%s", addr, key)
	req, err := http.NewRequestWithContext(reqCtx, http.MethodPut, url, bytes.NewReader(body))
	if err != nil {
		return 0, fmt.Errorf("building request to %s: %w", nodeID, err)
	}
	req.Header.Set("Content-Type", "application/json")

	resp, err := c.client.Do(req)
	if err != nil {
		return 0, fmt.Errorf("replicating to %s: %w", nodeID, err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		return 0, fmt.Errorf("replica %s returned status %d", nodeID, resp.StatusCode)
	}

	var respBody replicate.Response
	if err := json.NewDecoder(resp.Body).Decode(&respBody); err != nil {
		return 0, fmt.Errorf("decoding response from %s: %w", nodeID, err)
	}
	return respBody.Version, nil
}

A bug this design caught during testing, not after

The first version of write cancelled a shared context for the whole fan-out as soon as it returned — on the theory that once the outcome is decided, there's no reason to keep waiting on the rest. Running it against three real processes surfaced the problem immediately: a write with W=2 out of N=3 would return success, and the third replica would sometimes simply never get the value. Not "eventually" — never, because cancelling the shared context aborted its in-flight HTTP request mid-flight.

The fix is the detached context.Background() per replica goroutine shown above, instead of a context derived from the function's own lifetime. It matters for a reason worth being explicit about: ctx here ultimately comes from r.Context() in the HTTP handler, and Go's net/http cancels that context the instant the handler function returns — which is exactly when this function returns after reaching W. A context derived from it would have been cancelled by the standard library itself, independent of anything this code does, at precisely the moment it mattered most. This is a real, general trap: request-scoped contexts are correct for work that must finish before the response is written, and actively wrong for work meant to outlive the response.

Wiring it together

// internal/api/handler.go (PUT/DELETE, changed from Part 1)
func (h *Handler) handlePut(w http.ResponseWriter, r *http.Request) {
	key := r.PathValue("key")
	if key == "" {
		http.Error(w, "missing key", http.StatusBadRequest)
		return
	}

	body, err := io.ReadAll(r.Body)
	if err != nil {
		http.Error(w, "failed to read body", http.StatusBadRequest)
		return
	}
	defer r.Body.Close()

	result, err := h.coordinator.Put(r.Context(), key, body)
	if err != nil {
		writeCoordinatorError(w, err)
		return
	}

	w.Header().Set("X-Timestamp", strconv.FormatInt(result.Timestamp, 10))
	w.Header().Set("X-Acks", strconv.Itoa(result.Acks)+"/"+strconv.Itoa(result.Total))
	w.WriteHeader(http.StatusOK)
}

func writeCoordinatorError(w http.ResponseWriter, err error) {
	if errors.Is(err, coordinator.ErrQuorumUnreachable) {
		http.Error(w, err.Error(), http.StatusServiceUnavailable)
		return
	}
	http.Error(w, err.Error(), http.StatusInternalServerError)
}

DELETE follows the same shape, calling h.coordinator.Delete and returning 204 on success. GET is unchanged from Part 2 — it still reads only h.store, the local map, with no ring lookup at all.

// cmd/kvnode/main.go
func main() {
	configPath := flag.String("config", "cluster.json", "path to the cluster config file")
	nodeID := flag.String("id", "", "this node's ID, must match an entry in the cluster config")
	flag.Parse()

	if *nodeID == "" {
		log.Fatal("-id is required")
	}

	cfg, err := cluster.Load(*configPath)
	if err != nil {
		log.Fatalf("loading cluster config: %v", err)
	}

	self, err := cfg.Self(*nodeID)
	if err != nil {
		log.Fatalf("resolving self: %v", err)
	}

	ring := hashring.New(cfg.NodeIDs(), 0)

	s := store.New()
	coord := coordinator.New(self.ID, ring, cfg, s)

	kvHandler := api.NewHandler(s, coord)
	clusterHandler := api.NewClusterHandler(cfg, ring)
	replicaHandler := api.NewReplicaHandler(s)

	mux := http.NewServeMux()
	kvHandler.Routes(mux)
	clusterHandler.Routes(mux)
	replicaHandler.Routes(mux)

	log.Printf("node %s listening on %s (cluster size %d, replication factor %d, write quorum %d)",
		self.ID, self.Addr, len(cfg.Nodes), cfg.ReplicationFactor, cfg.WriteQuorum)
	if err := http.ListenAndServe(self.Addr, mux); err != nil {
		log.Fatal(err)
	}
}

cluster.json picks up the new field:

{
  "nodes": [
    {"id": "node-1", "addr": "localhost:8081"},
    {"id": "node-2", "addr": "localhost:8082"},
    {"id": "node-3", "addr": "localhost:8083"}
  ],
  "replication_factor": 3,
  "write_quorum": 2
}

Verifying it

Unit and race tests

The coordinator tests run against real httptest.Server instances for the remote replicas — not mocks that assert a function was called, actual HTTP servers wrapping actual ReplicaHandlers wrapping actual Stores — plus the local store directly for the replica that happens to be "self." That combination is what makes it possible to prove the properties that matter:

go test -race ./...

ok  	distributed-kv-store/internal/api
ok  	distributed-kv-store/internal/cluster
ok  	distributed-kv-store/internal/coordinator
ok  	distributed-kv-store/internal/hashring
ok  	distributed-kv-store/internal/store

Specific things the coordinator tests check, each backed by an assertion rather than a comment:

• TestPutReplicatesToAllThreeStoresWithSameTimestamp — after a successful Put, reads all three backing stores directly (bypassing the coordinator entirely) and confirms not just that the value is there, but that the Timestamp field is bit-for-bit identical across all three.

• TestPutSucceedsWithOneReplicaDown — closes one of the two remote httptest.Servers before calling Put, and confirms the write still succeeds with exactly 2 acks out of 3, because W=2.

• TestPutFailsWhenQuorumUnreachable — closes both remote servers, leaving only the local replica reachable, and confirms Put returns an error.

• TestFailedQuorumDoesNotRollBackReplicasThatAlreadyAcked — same setup as above, then checks the local store directly and confirms the value is there despite the overall call having returned an error. The coordinator doesn't undo work that already succeeded just because the aggregate outcome was failure.

• TestPutRejectsUpfrontWhenReplicasBelowWriteQuorum — constructs a config where W exceeds the number of replicas the ring can return, and confirms the coordinator errors out immediately, before attempting any network call at all.

gofmt -l ., go vet ./..., and go build ./... are all clean.

Three real processes, actually replicating

Same as Part 2, the properties that matter are the ones that hold across separate OS processes, not just within one test binary. Three nodes, started from the same cluster.json:

./kvnode -id node-1 &
./kvnode -id node-2 &
./kvnode -id node-3 &

A write coordinated through node-2, for a key whose replica set (per Part 2's ring) is {node-3, node-1, node-2}:

$ curl -i -X PUT localhost:8082/kv/order:777 -d "widget x3"
HTTP/1.1 200 OK
X-Acks: 2/3
X-Timestamp: 1786520288107510525

$ curl -s localhost:8081/kv/order:777
{"value":"widget x3","version":1,"timestamp":1786520288107510525}
$ curl -s localhost:8082/kv/order:777
{"value":"widget x3","version":1,"timestamp":1786520288107510525}
$ curl -s localhost:8083/kv/order:777
{"value":"widget x3","version":1,"timestamp":1786520288107510525}

All three processes — three separate binaries, three separate in-memory maps, no shared state whatsoever — hold the identical value with the identical timestamp. X-Acks: 2/3 confirms the coordinator only needed two of them before responding; the third caught up in the background, which is precisely the fix described above actually working, not just compiling.

Deleting through a different node (node-1 this time) propagates the same way:

$ curl -i -X DELETE localhost:8081/kv/order:777
HTTP/1.1 204 No Content
X-Acks: 2/3

$ curl -o /dev/null -w "%{http_code}\n" localhost:8081/kv/order:777
404
$ curl -o /dev/null -w "%{http_code}\n" localhost:8082/kv/order:777
404
$ curl -o /dev/null -w "%{http_code}\n" localhost:8083/kv/order:777
404

And the quorum boundary, checked by actually killing processes rather than describing what should happen: with node-3 killed (2 of 3 nodes left), a write to node-1 still succeeds —

$ curl -i -X PUT localhost:8081/kv/cart:9 -d "2 widgets"
HTTP/1.1 200 OK
X-Acks: 2/3

— and with node-2 also killed (only node-1 left, and W=2), the same kind of write fails, with the coordinator's own accounting in the error message:

$ curl -i -X PUT localhost:8081/kv/cart:10 -d "should fail"
HTTP/1.1 503 Service Unavailable

write quorum unreachable: got 1 acks (needed 2) out of 3 replicas

What this part does not guarantee

GET still only reads the local node. If a client writes order:777 through node-2 and then happens to GET it from a fourth node that isn't in that key's replica set, or queries a replica whose forwarded write is still in flight, it can get a 404 for a key that a quorum write just placed elsewhere in the cluster, seconds ago. Nothing here makes a node consult the ring before answering a read. That's the entire subject of Part 4.

A write reporting success means W replicas have it — it does not mean all N do, even though this part's fix means the rest are actively trying to catch up rather than being silently dropped. If a replica is down for the whole duration of a write, not just slow, it never gets that write at all, and nothing currently notices or retries later. A read that happens to land on that replica before it's repaired (Part 5) or handed the missed write (Part 6) will see stale or missing data. This part makes the common case durable; it doesn't yet make every case eventually consistent.

If the coordinator process itself crashes partway through a fan-out — after some replicas have acked but before it can respond to the client — the client gets a connection error with no way to know how many replicas actually got the write. Those replicas keep whatever they were given; nothing rolls back. This is the same "partial success, reported as total uncertainty" shape as the quorum-failure case tested above, just triggered by the coordinator dying instead of replicas being unreachable. Nothing added in this part addresses it, and nothing planned for the rest of the series specifically targets coordinator crashes either — it's a structural consequence of not having a leader or a shared log, worth naming rather than leaving implicit.

Part 4 is what makes reads trustworthy: consulting the same replica set a write would have used, sending the read to multiple replicas, and picking the newest version among the responses instead of trusting whichever single node happened to receive the request.