Y

Microservices · Intermediate

Service boundaries, events and the outbox pattern

4 min read

Why it matters

More than half of the Software Engineer job descriptions we read mention microservices. Splitting a system is easy; keeping it correct is not. The two classic bugs are an event that is lost because the service crashed between saving data and publishing, and an event that is processed twice because the broker redelivered it.

The concept

Boundaries. Split by business capability (orders, payments, catalogue), not by technical layer. Each service owns its data; other services never read its tables directly. If two services must always change together, they probably belong together. Many teams start with a well-structured monolith and split only the parts that need independent scaling or release cycles.

Communication.

  • Synchronous (HTTP or gRPC): the caller waits. Simple, but the caller fails when the callee fails, and latency adds up along a chain.

  • Asynchronous (events through Kafka, RabbitMQ or a cloud queue): the producer publishes "order placed" and carries on; interested services react. Services are decoupled, but the system becomes eventually consistent.

The dual-write problem. "Save the order, then publish the event" fails if the process crashes in between. The outbox pattern writes the event to an outbox table in the same database transaction as the order; a relay process then publishes rows from the outbox.

At-least-once delivery. Brokers and relays retry, so consumers will sometimes see an event twice. Make consumers idempotent: record processed event ids and ignore repeats.

Worked example: orders and loyalty points

outbox.py
import json
import sqlite3

db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT, amount INTEGER, status TEXT);
CREATE TABLE outbox (id INTEGER PRIMARY KEY, topic TEXT, payload TEXT, published INTEGER DEFAULT 0);
CREATE TABLE processed_events (event_id INTEGER PRIMARY KEY);
CREATE TABLE loyalty_points (customer TEXT PRIMARY KEY, points INTEGER);
""")


def place_order(customer: str, amount: int) -> int:
    # The order and the event are written in ONE transaction: both happen or neither does.
    with db:
        order_id = db.execute("INSERT INTO orders (customer, amount, status) VALUES (?, ?, 'placed')", (customer, amount)).lastrowid
        db.execute("INSERT INTO outbox (topic, payload) VALUES ('order.placed', ?)",
                   (json.dumps({"order_id": order_id, "customer": customer, "amount": amount}),))
    return order_id


def relay() -> list[tuple[int, dict]]:
    # A separate process reads unpublished rows and sends them to the broker (here: returns them).
    rows = db.execute("SELECT id, payload FROM outbox WHERE published = 0 ORDER BY id").fetchall()
    with db:
        db.executemany("UPDATE outbox SET published = 1 WHERE id = ?", [(r[0],) for r in rows])
    return [(event_id, json.loads(payload)) for event_id, payload in rows]


def handle_order_placed(event_id: int, event: dict) -> None:
    # Idempotent consumer: remembers event ids, so a redelivered event changes nothing.
    with db:
        if db.execute("INSERT OR IGNORE INTO processed_events VALUES (?)", (event_id,)).rowcount == 0:
            return
        db.execute("INSERT INTO loyalty_points VALUES (?, ?) ON CONFLICT(customer) DO UPDATE SET points = points + excluded.points",
                   (event["customer"], event["amount"] // 100))


place_order("asha", 2499)
place_order("asha", 1250)
events = relay()
for event_id, event in events + events[:1]:  # the broker redelivers the first event
    handle_order_placed(event_id, event)

print("events delivered:", len(events) + 1)
print("points:", db.execute("SELECT * FROM loyalty_points").fetchall())
shellsession
$ python3 outbox.py
events delivered: 3
points: [('asha', 36)]

Asha earns 24 points for the first order and 12 for the second. The first event arrived twice, but the consumer recorded its id the first time and ignored the repeat; without that check she would have 60 points. In a real system the services have separate databases: the outbox lives in the orders database and processed_events in the loyalty database, and the relay is a separate process or a change-data-capture tool such as Debezium.

Common mistakes

  • A distributed monolith. Services that call each other synchronously for every request and must be deployed together. You pay the costs of distribution and get none of the benefits.

  • Shared databases between services. A schema change in one breaks the other.

  • Assuming exactly-once delivery. Design for duplicates and out-of-order events.

  • Splitting too early. Boundaries are expensive to move once there is a network between them.

Interview angle

"When would you not use microservices?", "how do you keep data consistent across services?" (outbox, sagas with compensating actions, eventual consistency), "what is idempotency and how do you implement it for a consumer?", and "synchronous or asynchronous for this flow?". Draw the flow and point at where things can fail.

Practice task

  1. Add a payments consumer for the same event that marks the order paid and publishes payment.succeeded through its own outbox.

  2. Simulate a crash in relay after publishing but before marking rows as published, and show that the idempotent consumer still produces the right points.

Further reading