Automated testing, for beginners

Why bother in a four-hour hackathon?

A test is a small program that checks your program. You write it once and it re-checks your work every time anything changes, in seconds.

With an AI agent, tests do double duty: Claude can run them itself after every edit and fix what it broke before you ever see it. Asking for tests is the single best way to make an agent's work trustworthy.

For civic data they matter even more. A chart that is off by a factor of ten looks exactly as convincing as a correct one.

The kinds of test

Many small fast tests at the bottom, a few slow thorough ones at the top.

Your first test: check the data

Put this in tests/test_trees.py, run python -m pytest, and watch it pass or tell you exactly what is wrong. Or ask Claude: write data tests like these for our dataset.

import pandas as pd
import pytest

@pytest.fixture(scope="module")
def trees():
    return pd.read_csv("data/trees-oak-park.csv")

def test_file_is_not_empty(trees):
    assert len(trees) > 1000

def test_expected_columns_exist(trees):
    # Ask Claude which columns your analysis relies on, then list them here.
    for column in trees.columns[:3]:
        assert trees[column].notna().any(), f"{column} is entirely blank"

def test_no_exact_duplicate_rows(trees):
    assert not trees.duplicated().any()

Let Claude test its own work

The pattern is called test-driven development: describe what right looks like, let the test fail, then let Claude make it pass.

1. Describe the behaviour

   Write a pytest test for a function share_by_genus(df) that returns each genus's share of all trees, summing to 100%. Don't write the function yet.

2. Watch it fail

   Run the test and show me it fails. A failing test proves the test actually checks something.

3. Make it pass

   Now write the function until the test passes. Don't change the test.

4. Keep it running

   Add a line to CLAUDE.md: Run python -m pytest before every commit. Claude will check its own work from then on.

End-to-end: a robot clicks your page

Install with pip install pytest-playwright then playwright install chromium. Or add the Playwright connector from the toolbox and just ask Claude to click around.

from playwright.sync_api import Page, expect

def test_map_page_loads(page: Page):
    page.goto("http://localhost:8000/")
    expect(page.get_by_role("heading", level=1)).to_be_visible()
    expect(page.locator(".leaflet-container")).to_be_visible()

def test_works_on_a_phone(page: Page):
    page.set_viewport_size({"width": 375, "height": 740})
    page.goto("http://localhost:8000/")
    # Nothing should be wider than the screen.
    assert page.evaluate("document.documentElement.scrollWidth") <= 375

Run every test on every push

Save this as .github/workflows/ci.yml in your repository. GitHub runs your tests each time you push, and shows a green tick or a red cross on every commit.

name: CI
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install pandas pytest
      - run: python -m pytest -v

A real example: this site

The showcase you are reading is tested the same way. Browse its tests for working examples.