Dedy
Foundations

Lesson 2. Engineered prompts and evaluation sets

Write prompts like specifications, then prove them with numbers.

Level: Foundations · Time: 4 hours · Output: an evals/ folder reused throughout the course

Objectives

  • Write prompts as specifications: role, task, constraints, output format, examples.
  • Build an evaluation set with expected answers and automatic grading, instead of "it reads fine".
  • Use an LLM as a judge under control, and measure the judge's own bias.
  • Compare two prompt versions with numbers, and know when a difference is just noise.

A prompt is a specification

A good prompt answers the five questions a new colleague would also need answered:

Component Question Short example
Context Who uses the output, and for what? "Shown in a banking app to elderly customers"
Task What exactly must be done? "Classify the message into one of six labels"
Constraints What must never happen? "Never guess a balance; if data is missing, return needs_human"
Format How will a machine read it? JSON matching a schema, nothing else
Examples What does a hard case look like? Three to five examples, at least one edge case

The techniques with the best return on effort, in order: clear instructions, well-chosen examples (few-shot), separating data with tags such as <document>, letting the model think before answering on multi-step tasks, and enforcing format with a schema or structured output. Do not "shout" at the model in capitals; contradictory and vague instructions are the main cause of errors.

Evals: what separates an engineer from someone trying their luck

An eval set has three parts: test cases (input and expected answer), a grader, and a report. Graders in order of preference:

  1. Exact or rule-based match (labels, numbers, valid JSON): cheap, deterministic, use them as much as possible.
  2. Code-based logic (contains required keywords, stays under a length, calls the right function).
  3. LLM as judge for soft criteria (tone, completeness). It needs a clear rubric, a small scale or pass/fail, and its agreement with a human grader must be measured on 20 to 30 samples before you trust it.

Classic judge traps: preferring longer answers, preferring whichever answer comes first, and preferring its own style. Mitigations: swap the order in pairwise comparisons, require quoted evidence before a score, and use a judge from a different model family where possible.

Minimum statistics. With 50 cases and around 80% accuracy, the standard error is roughly sqrt(0.8 × 0.2 / 50) ≈ 5.7 percentage points. A prompt that is "3% better" on 50 cases may well be noise. Add cases, run paired comparisons case by case, or only conclude when the gap is large enough.

Lab: an eval set for customer messages

Invented setting: an online bakery, "Cloud Cakes", receives chat messages. Labels: order, cancel, price, complaint, delivery, other.

  1. Write 30 messages yourself, at least four per label, including six hard cases: two intents in one message, slang, missing punctuation, sarcasm ("lovely cake, only took three hours to arrive").
  2. Save them to evals/cloud_cakes.jsonl, one {"id":..., "input":..., "label":...} per line.
  3. Write run_eval.py: call the model with prompt v1, parse JSON, compare labels, print accuracy and a confusion matrix.
import json, collections
from anthropic import Anthropic
client = Anthropic()
PROMPT = open("prompts/v1.txt").read()
cases = [json.loads(l) for l in open("evals/cloud_cakes.jsonl")]
confusion, ok = collections.Counter(), 0
for c in cases:
    r = client.messages.create(model="claude-haiku-4-5", max_tokens=50, temperature=0,
        system=PROMPT, messages=[{"role": "user", "content": c["input"]}])
    pred = json.loads(r.content[0].text)["label"]
    ok += pred == c["label"]; confusion[(c["label"], pred)] += 1
print(f"accuracy {ok}/{len(cases)}"); print(confusion.most_common(10))

The model name is illustrative; take the current name from your provider's Models page.

  1. Write prompt v2 that fixes exactly the errors the confusion matrix reveals. Run again and record both versions in RESULTS.md.

Challenges

Easy (10 points). Prompt v2 reaches at least 90% on the 30 cases and always returns valid JSON (zero parse errors). Acceptance: run log and prompt file.

Medium (20 points): a verified judge. Add a reply field (the answer to the customer) and grade it with an LLM against a three-criterion pass/fail rubric. Grade 25 samples by hand. Acceptance: agreement between you and the judge ≥ 80%, plus one concrete case the judge got wrong and how you changed the rubric.

Boss (40 points): attack your own eval. Grow the set to 120 cases, 40 of them generated by a different LLM specifically to break your prompt (you filter and relabel them). Acceptance: accuracy with a 95% bootstrap confidence interval, the three most common error types, and proof that the final prompt beats v1 with a paired test (for example McNemar) at p < 0.05.

Self-check quiz

  1. Why use rule-based graders before an LLM judge?
  2. Name two common LLM-as-judge biases and how to reduce them.
  3. Prompt A scores 41/50, prompt B 43/50. Should you conclude B is better?
  4. How should few-shot examples be chosen?
  5. Why does wrapping input documents in XML tags help?

Answers

  1. They are cheap, fast, deterministic and unbiased; an LLM judge itself needs verifying.
  2. Length bias and position bias; reduce them with an evidence-first rubric, swapping order in pairwise comparisons, and checking against human grades.
  3. Not yet. A 4-point gap lies inside the roughly ±6-point margin of 50 cases; add cases or use a paired test.
  4. Varied, in the exact output format, including edge cases, and never copied from the test set, so you do not "memorise" the eval.
  5. It cleanly separates instructions from data, reduces confusion, and is a first line of defence against prompt injection (Lesson 7).

Further reading