Y

Project · Advanced · about 28 hours

A data question agent with guardrails and an eval suite

Build a tool-using agent that answers questions about NYC taxi trips with read-only SQL, cites the query it ran, stays within budgets and is measured on a task suite.

Skills:AI agents and tool use, LLM evaluation, Monitoring and observability, Python, SQL

Part of: AI/ML Engineer, Forward Deployed Engineer

Data and licences

Brief

Analysts want to ask plain-language questions such as "what share of January trips from JFK paid by card?". An agent with a SQL tool can answer, but it can also run a query that scans everything, invent a column or report a number it never computed. Build one that is useful and safe.

Load one month of Yellow Taxi parquet data into DuckDB (a few million rows). The data is public, so the risks are cost, correctness and trust rather than privacy.

What to build

  1. Tools: describe_schema(), run_sql(query) (read-only connection, SELECT only, row limit, timeout) and make_chart(query, kind).

  2. Run the agent loop with the Anthropic API tool runner or your own loop, with a maximum number of steps and a token budget.

  3. Every answer must include the SQL it ran. Reject answers that state numbers not present in a tool result.

  4. Require human approval before exporting more than 1,000 rows.

  5. Write 40 questions with expected answers computed by your own SQL. Include ambiguous questions (the agent should ask) and out-of-scope questions (it should refuse).

  6. Grade runs automatically and record steps, tokens, latency and failure category for each task. Run the suite three times to see how much results vary.

Deliverables

  • Repository with agent/, tools/, tests/ (one test per guardrail), evals/tasks.jsonl, evals/report.md and trace samples

How it is graded

Reviewers try to make the agent drop a table, scan the full file without filters and invent a number. Then they run your suite and compare the report with the traces.

Stretch

Expose the same tools through an MCP server and show the agent works through either path.

Deliverables

  • An agent with at least three tools (list tables and columns, run a read-only query, make a chart)
  • Guardrails for SQL safety, row and time limits, step and cost budgets, and a human approval step for exports
  • A task suite of at least 40 questions with expected answers, graded automatically
  • Traces for every run and a report on accuracy, steps, cost and failure types

How a reviewer grades it

CriterionWeightWhat good looks like
Tools are narrow and well described; the agent never gets raw write access20%
Guardrails are enforced in code, and tests show each one stopping a bad action25%
Task suite with automatic grading and a baseline run30%
Traces and a failure analysis a team could act on25%