Y

Project · Intermediate · about 20 hours

A cited question-answering assistant over the Kubernetes docs

Build retrieval over the official Kubernetes documentation, answer questions with citations, and measure retrieval and answer quality on your own test set.

Skills:LLMs and NLP, LLM evaluation, Python, Retrieval-augmented generation (RAG)

Part of: AI/ML Engineer, Forward Deployed Engineer

Data and licences

Brief

Platform teams answer the same Kubernetes questions again and again. The official docs are open (CC BY 4.0) and change often, which makes them a good corpus for a retrieval-augmented assistant that must cite its sources and say "I don't know" when the docs are silent.

What to build

  1. Clone kubernetes/website and read the Markdown under content/en/docs. Strip front matter and shortcodes, keep headings.

  2. Chunk by section. Try two chunk sizes and record how many chunks each produces.

  3. Embed with a small open embedding model and store vectors with the file path and heading.

  4. Write 60 or more test questions yourself, each with the page that answers it. Include 10 questions the docs do not answer.

  5. Measure recall at 5 for both chunkings. Add BM25 and try a hybrid of the two.

  6. Answer with an LLM using only retrieved chunks, citing chunk ids. Check that every citation exists and supports the answer. Refuse when retrieval scores are low.

  7. Report: recall at k, citation accuracy on a sample you grade by hand, refusal rate on the unanswerable questions, latency and cost per question.

Deliverables

  • Repository with ingest/, service/, eval/questions.jsonl, eval/report.md and attribution to the Kubernetes docs

How it is graded

Reviewers run your evaluation and ask ten questions of their own. The key question: does it find the right page, and does it admit when it cannot?

Stretch

Re-ingest only changed files after a git pull, and show that the evaluation still passes.

Deliverables

  • An ingestion script that chunks the Markdown docs and stores embeddings with source paths
  • An answer service that retrieves, answers with citations and refuses when the docs do not cover the question
  • A test set of at least 60 questions with the documents that answer them
  • An evaluation report on retrieval recall, citation accuracy and refusals

How a reviewer grades it

CriterionWeightWhat good looks like
Chunking keeps headings and source paths, and the choice is justified with numbers20%
Retrieval measured with recall at k on the test set, before any answer quality work25%
Answers cite the chunks they use, and unsupported questions are refused30%
Report is honest about failure cases and cost per question25%