← The Reviewer’s Eye

Module 1: The Reviewer’s Eye

Why judgment is the scarce skill — and how to train it.

Production is solved. Oversight isn’t.

For decades the valuable skill was production — writing the code, drafting the policy, building the model. AI now produces a credible first draft of all of it, in seconds, at volume. The bottleneck moved. The scarce skill is no longer producing output; it is judging it: inspect → validate → correct → escalate → approve.

That is a perceptual skill, not a knowledge one. You probably already know what SQL injection is. The hard part is seeing it in the third file of a 600-line diff at 4pm — fast, reliably, without flagging the ten safe queries next to it.

A wrong answer from a capable model is syntactically clean, well-formatted, and confident, and missing the one control that matters. Fluency is exactly what makes bad output dangerous. Train your eye on the shape of badness, not the vocabulary.

This course is reps, not lectures

Each module pairs short readings with drills: real AI-generated artifacts you judge under a timer — Ship, Flag, or Escalate — then immediate feedback on the critical cue you should have caught and the distractor that made bad output look fine.

  • Ship — you’re confident it’s safe to merge.
  • Flag — there is a definite problem; block it.
  • Escalate — you can’t verify the claim alone; route it to a human/source before acting.

Calibrated trust, not paranoia

A reviewer who flags everything gets ignored — alert fatigue is a security failure, not safety. A reviewer who ships everything is worse. The goal is the narrow skill in between, measured on two axes at once:

  • Catch rate — of the dangerous artifacts, how many did you stop?
  • False-alarm rate — of the safe artifacts, how many did you wrongly block?

Most people are strong on one axis and blind on the other. The drills deliberately mix safe-but-scary cases — a raw SQL string that’s actually parameterized, a subprocess call that’s actually a list with no shell — so that "flag everything" loses points.

The 4-pass scan

  1. Trust boundary — where does external input enter? (request args, body, headers, filenames, URLs)
  2. Sink — where does that input go? (a query, a shell, a deserializer, an HTML page, a fetch)
  3. Control — is there a real control between them? (parameterization, allow-list, ownership check)
  4. Claim — if the artifact asserts something (a CVE, a fix, "no changes needed"), can you verify it?
Throughout, watch the model itself: it will sometimes over-flag safe code and sometimes confidently approve a vulnerability. Your job is to overrule it in both directions.

Ready? Hit the warm-up drill below. Four cards, two of them safe. See where your eye is right now.

Drill — The Reviewer’s Eye

Judge each artifact. You are scored on calibration: catches vs false alarms.

CARD 1 / 4catches 0/0false alarms 0/0
AI-generated · Python / FlaskSecure Code Review
Prompt to the AI » "Add a search endpoint that filters items by a query string."
@app.get("/search")def search():    term = request.args.get("q")    sql = f"SELECT * FROM items WHERE name LIKE '%{term}%'"    return jsonify(db.execute(sql).fetchall())

Capstone: the Capstone: Calibration Exam (16 cards) from The Reviewer’s Eye unlocks once modules are complete.