Skills Night › Introduction to AI

Lesson 5.5 — Evals: Proving Your Agent Works

Introduction to AI · Module 5: AI Agents · About 75 min · Windows PC, PowerShell 5, Claude Code (Max)

Goal: stop eyeballing. Run your receipts agent on 5 test cases × several runs, get a pass rate, change one instruction, and measure again.

An eval is a repeatable test with a clear pass/fail. Agents can vary from run to run, so you run each case more than once. Module 11 goes deeper.

Before you start

Step 1 · Save the task as a file (⏱ 0:00–0:05)

cd $HOME\agent-night
@'
Read every file in receipts\. Create out\receipts.csv with columns date,store,category,total
(categories: Groceries, Fuel, Home, Utilities, Health) and out\summary.md (one page: total,
per category, biggest purchase, one tip). Run .\check-receipts.ps1 and fix until PASS.
Never edit or delete files in receipts\.
'@ | Out-File task.txt -Encoding utf8

Step 2 · Build 5 test cases (⏱ 0:05–0:20)

Each case is a twist on the receipts. Paste the whole block:

mkdir cases -Force | Out-Null
foreach ($n in 1..5) { Remove-Item "cases\c$n" -Recurse -ErrorAction SilentlyContinue; Copy-Item receipts "cases\c$n" -Recurse }
# c1 = normal. c2 = one receipt uses lowercase "total aud"
(Get-Content cases\c2\r03.txt) -replace "TOTAL AUD", "total aud" | Out-File cases\c2\r03.txt -Encoding utf8
# c3 = a receipt with extra noise lines
Add-Content cases\c3\r05.txt "Thank you for shopping!`r`nABN 12 345 678 901"
# c4 = a date in another format
(Get-Content cases\c4\r07.txt) -replace "2026-10-15", "15/10/2026" | Out-File cases\c4\r07.txt -Encoding utf8
# c5 = a junk file that isn't a receipt
"Shopping list: milk, eggs" | Out-File cases\c5\notes.txt -Encoding utf8
dir cases

You should see folders c1–c5.

Case 5 contains a non-receipt, so the right answer is still 10 rows. The checker expects exactly that.

Step 3 · The eval runner (⏱ 0:20–0:30)

@'
param([int]$Runs = 2)
$results = @()
foreach ($n in 1..5) { foreach ($r in 1..$Runs) {
  Remove-Item receipts, out -Recurse -ErrorAction SilentlyContinue
  Copy-Item "cases\c$n" receipts -Recurse
  $t = Get-Content task.txt -Raw
  claude -p $t --allowedTools "Read,Write,Edit,Glob,PowerShell,Bash" | Out-Null
  $res = (.\check-receipts.ps1 | Select-Object -Last 1)
  Write-Host "case $n run $r : $res"
  $results += [pscustomobject]@{case=$n; run=$r; result=$res}
}}
Remove-Item receipts -Recurse; Copy-Item cases\c1 receipts -Recurse
$pass = @($results | ? result -eq "PASS").Count
"PASS RATE: $pass / $($results.Count)"
$results | Export-Csv ("eval-" + (Get-Date -Format "HHmm") + ".csv") -NoTypeInformation
'@ | Out-File run-evals.ps1 -Encoding utf8

Step 4 · Baseline run (⏱ 0:30–0:50)

.\run-evals.ps1 -Runs 2

You should see 10 lines like case 1 run 1 : PASS, then e.g. PASS RATE: 8 / 10. It takes 10–20 min, so take a break.

Step 5 · Change ONE thing, re-run (⏱ 0:50–1:10)

Pick the case that failed most. Add one line to task.txt that addresses it, e.g.:

Ignore files that are not receipts. Convert all dates to YYYY-MM-DD.
notepad task.txt
.\run-evals.ps1 -Runs 2
Pass rateFailing cases
Before
After

✅ Pass check

Troubleshooting

ProblemFix
Every run FAILs instantlyRun one case by hand: claude -p (Get-Content task.txt -Raw) and read what it says.
Hit a usage limitUse -Runs 1, or finish after the limit resets.
Running scripts is disabledSet-ExecutionPolicy -Scope CurrentUser RemoteSigned, then Y.
Messed up receipts folderThe runner restores c1 at the end. Or re-run Step 1 of Lesson 5.4.
Stuck? Ask Skills Coach. Send a screenshot or paste the error plus the step number. Don't stay stuck on one step for more than 10 minutes.

Finish this lesson

Back to all modules and lessons

← Previous (5.4)