Designing agents you can trust

#Why it matters

Agents can take actions: send emails, change records, run code. When they go wrong, they go wrong with your permissions. Forward deployed and AI engineers are hired to ship these systems to customers safely, and interviews probe the failure modes: loops, wrong tool calls, prompt injection through tool results, and runaway cost.

#The concept

Start simple. Many tasks are better as a fixed workflow (step 1, step 2, step 3 in code, with the model filling in each step) than as an open-ended agent that decides its own steps. Use an agent when the path genuinely varies by input and the value justifies the extra cost and risk.

Guard rails for agents:

• Least-privilege tools: narrow actions (refund_order(order_id, amount) with a cap) rather than general ones (run_sql, http_request).

• Human approval for irreversible or high-value actions, enforced in the tool code, not only in the prompt.

• Limits: iterations, tokens, time and money per task.

• Untrusted inputs: documents, web pages and tool results can contain instructions. Treat them as data; never let them expand what the agent may do.

• Observability: trace every model call and tool call with inputs, outputs, duration and cost.

• Evaluation: a set of realistic tasks with checks on the outcome and on the path (did it call the refund tool at most once?).

#Worked example: an approval gate inside the tool

A support agent may refund orders, but anything above 2,000 rupees needs a human:

from dataclasses import dataclass

from anthropic import beta_tool

APPROVAL_LIMIT_INR = 2000
PENDING: list[dict] = []          # in production: a queue or ticket system


@dataclass
class Order:
    order_id: str
    amount_inr: int
    refunded_inr: int = 0


ORDERS = {"A-1001": Order("A-1001", 1450), "A-1002": Order("A-1002", 8999)}


@beta_tool
def refund_order(order_id: str, amount_inr: int, reason: str) -> str:
    """Refund part or all of an order. Use only after confirming the order id and the reason with the customer.

    Args:
        order_id: The order id, for example A-1001.
        amount_inr: Amount to refund in rupees. Cannot exceed what is left to refund.
        reason: One sentence explaining why, shown to the finance team.
    """
    order = ORDERS.get(order_id)
    if order is None:
        return f"Error: order {order_id} not found."
    remaining = order.amount_inr - order.refunded_inr
    if not 0 < amount_inr <= remaining:
        return f"Error: amount must be between 1 and {remaining} rupees for this order."
    if amount_inr > APPROVAL_LIMIT_INR:
        PENDING.append({"order_id": order_id, "amount_inr": amount_inr, "reason": reason})
        return (f"Refund of {amount_inr} rupees is above {APPROVAL_LIMIT_INR} and has been sent for human approval. "
                "Tell the customer it will be reviewed within one working day.")
    order.refunded_inr += amount_inr
    return f"Refunded {amount_inr} rupees on {order_id}."

from refund_tool import PENDING, refund_order

print(refund_order.func("A-1001", 500, "Two items missing"))       # Refunded 500 rupees on A-1001.
print(refund_order.func("A-1001", 5000, "Duplicate"))              # Error: amount must be between 1 and 950 rupees ...
print(refund_order.func("A-1002", 8999, "Order never delivered"))  # ... sent for human approval ...
print(PENDING)

The limit holds whatever the model is told or tricked into saying, because the check is in code. The error messages are written for the model: they say exactly what was wrong so it can recover or explain it to the customer.

Checklist before launch:

#Common mistakes

• Enforcing policy only in the prompt. "Never refund more than 2,000" in a prompt is a suggestion; in code it is a rule.

• Generic power tools. run_shell or http_get(any_url) turn every prompt injection into a real attack.

• No traces. When a customer complains, you need to see exactly what the agent did and why.

• Launching without an evaluation set. You cannot tell whether a prompt change helped or hurt.

#Interview angle

"Design an AI agent that handles refunds for an e-commerce company." Discuss whether a workflow suffices, the tool set and its limits, approval thresholds in code, injection risks from customer messages, logging and tracing, evaluation tasks with expected actions, and a gradual rollout with human review of samples.

#Practice task

1. Add an idempotency key to refund_order so the same refund request cannot run twice.

2. Write ten evaluation tasks for the refund agent, including two that try to talk it into an unapproved refund, and the expected tool calls for each.

3. Rewrite the data question agent from A data question agent with real tools as a fixed workflow (plan SQL, run, summarise) and compare reliability on five questions.

#Further reading

• Anthropic engineering: building effective agents

• OWASP Top 10 for LLM applications

• Claude docs: tool use overview