Don’t Let Claude Grade Its Own Homework

I had a handful of parallel Claude agents write me up a strategy folder for a side project. The pull request looked great: ten documents, all neatly cross-linked, and it read well. Then the automated reviewer posted its findings, and Claude, to its credit, did not object:

The full audit came back: 30 of the 95 relative links pointed at files that did not exist. ./market.md instead of market-analysis.md, ./monetization.md instead of business-model.md, some files were even made up entirely! Skimming that PR as a human, I would have approved it, since every one of them looked plausible. Luckily, my automated reviewer caught it because it resolved every link instead of judging whether the names “vibed” right.

To be fair, Claude does not have any bad intent behind this. A language model simply generates the most plausible continuation, and sometimes the most plausible continuation is fiction delivered with complete confidence. From the reading side, though, that distinction brings little comfort. Confident and wrong reads exactly like confident and right.

This post is about the system I have in place to help me review Claude’s mistakes and hallucinations: a Codex agent running in GitHub Actions that reviews every pull request. It walks you through my setup, why cross-provider review beats both self-review and a human-reads-everything review, how to enforce memory and accountability in the review loop, the cost and reliability learnings I gathered along the way, and the actual code I use to make this work.

---

Agent throughput broke the human review loop

Hallucinations by themselves are a manageable problem, since a very careful review or functional assessment would catch them. However, what broke over the past year is the volume that needs reviewing. A single engineer now runs multiple agent sessions in parallel, and each of them can produce a thousand-line diff in the time it takes to refill your coffee. Producing code has become the fast and easy part; reviewing it honestly is what eats the day. That makes the human reviewer the bottleneck of the whole delivery pipeline, and the bottleneck is always the next thing you should automate.

That bottleneck invites two familiar failure modes. Either you approve whatever the agents produce without truly reading it, and the broken links, phantom parameters, and subtly wrong edge cases ship to production. Or you still insist on reading every line yourself, and your fast and smart AI agents work even less than you do. The first is negligent, the second wasteful, and most teams oscillate between both depending on the deadline.

The conclusion I keep landing on: our review process has to scale with the speed we’re producing code. The human element stays, but the human reviewer’s focus should go to the parts that truly matter. The grind of checking whether links resolve, whether the new parameter exists, whether there’s a bug or vulnerability, and whether the code changes match what was initially requested, all of that has to run automatically at the same cadence as the coding agents, even before a person ever spends a second reviewing the code.

Insight: hallucinations are a constant, throughput is what changed. Any process that requires a human to read every line caps your agents at human reading speed, so the very first review pass has to be automated.

---

Pick a reviewer that does not share the author’s blind spots

The obvious first idea to prevent these mistakes is to have the agent review its own work… but it is also the weakest one. A hallucination that made it into the diff is, by definition, plausible to the model that produced it. Asking the same weights to find mistakes in the code, especially within the same already-biased coding session, usually reproduces the exact blind spots that let the mistakes through.

This intuition has research behind it. Panickssery et al. showed at NeurIPS 2024 that LLM evaluators recognize and favor their own generations: the better a model is at identifying its own output, the more generously it scores it. Follow-up work ties this self-preference bias to familiarity: AI judges score text higher when it reads as predictable to them, and nothing reads as more predictable to a model than the text it just wrote itself.

Different LLM providers train on different data, with different recipes and different feedback loops. No provider manages to eliminate mistakes, but the kind of mistakes a model makes traces back to how it was trained, and that differs per provider. That decorrelation is exactly what you want in your review agent. In the year that I’ve used this setup, OpenAI’s Codex has consistently flagged issues in Claude-written PRs that Claude’s own review passed without blinking, from invented filenames to entire code blocks that had nothing to do with the initial request, or that introduced outright security violations.

One conclusion you cannot take from this is that Codex is the better engineer and you should hand it the keyboard instead of Claude. Codex misfires fairly regularly, too! Just in a different direction than Claude does. The power sits in working multimodel: combining providers within the same workflow, where the models fill each other’s gaps.

Another incorrect conclusion is that Codex’s reviews are perfect and can be accepted blindly. Hallucinations happen in reviews too, so treat the review as advisory. Challenging the author (be it Claude or yourself) puts the focus on potential blind spots, which is the perfect signal to zoom in and reflect. It remains the author’s job to own the final quality, and sometimes that means writing back to Codex that its review was wrong. More on this mechanism later in the post.

Insight: an author’s bugs are by definition plausible to the author, and research confirms LLMs are positively biased toward their own output. Review value comes from decorrelation, by using a model from a different provider that fails differently, and different is what catches what you miss.

---

The pipeline: one LLM call wrapped in ordinary engineering

Conceptually, an LLM reviewer can be written in a single line: “here is the code diff, please review it”. Both OpenAI and Anthropic ship hosted shortcuts for exactly that: OpenAI has a GitHub review integration that auto-reviews PRs, and Anthropic has claude-code-action. They are fine starting points, but I prefer to run my own action instead, because I want to own the prompt, the review lifecycle, the merge gating, and the model pin. All these aspects matter when your day-to-day becomes managing and reviewing agents.

So what am I using then, exactly? Two files that live in the repository they review:

.github/
├── workflows/
│   └── pr.yml            # orchestration: triggers, context, retries, posting
└── actions/
    └── codex-pr-review/
        └── action.yml    # the reviewer: pinned Codex call + review prompt

The runtime flow is short. Every time a PR opens or receives a push, pr.yml checks out the code, collects the PR’s full conversation history, and hands both to the composite action. The action runs Codex over the diff and returns its findings as a single message. The workflow then posts that message as a single, continuously updated comment in the PR conversation, and publishes a commit status that stays red while findings remain unresolved: the go/no-go signal sitting next to the rest of CI.

At the center of it all sits one deceptively simple step, the only line that actually touches an LLM:

- uses: openai/codex-action@v1
  with:
    openai-api-key: ${{ inputs.openai-api-key }}
    prompt: |
      # the review contract: what to review, how to track findings,
      # and how to report back (unpacked in the next section)

Everything else exists to run that single step reliably, and to make sure its feedback actually lands on the PR. Here is the complete workflow:

name: PR Reviewer

on:
  pull_request:
    branches: [dev] # all work rides feature branches: feature -> dev -> main
    types: [opened, synchronize, reopened]

# a new push cancels the in-flight (expensive) review of the previous commit
concurrency:
  group: pr-reviewer-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  codex:
    if: github.event.pull_request.user.login != 'dependabot[bot]'
    runs-on: ubuntu-latest
    timeout-minutes: 15 # normal review takes a few minutes, so 15m is more than enough
    permissions: # the token can read code and post feedback, but can never push or merge
      contents: read
      issues: write
      pull-requests: write
      statuses: write
    steps:
      # the /head ref updates synchronously on every push;
      # the auto-generated merge ref can race the checkout
      - uses: actions/checkout@v7
        with:
          ref: refs/pull/${{ github.event.pull_request.number }}/head

      # make sure both diff endpoints exist locally, so Codex can resolve base...head
      - name: Pre-fetch base and head refs for the PR
        run: |
          git fetch --no-tags origin \
            ${{ github.event.pull_request.base.ref }} \
            +refs/pull/${{ github.event.pull_request.number }}/head

      # feedback lives on three surfaces: issue comments, reviews, and inline comments
      - name: Fetch PR conversation
        id: conversation
        uses: actions/github-script@v9
        with:
          result-encoding: string
          script: |
            const prNumber = context.payload.pull_request.number;
            const opts = { owner: context.repo.owner, repo: context.repo.repo };
            const [comments, reviews, inline] = await Promise.all([
              github.paginate(github.rest.issues.listComments,
                { ...opts, issue_number: prNumber }),
              github.paginate(github.rest.pulls.listReviews,
                { ...opts, pull_number: prNumber }),
              github.paginate(github.rest.pulls.listReviewComments,
                { ...opts, pull_number: prNumber }),
            ]);
            const all = [
              ...comments.map(c => ({
                date: c.created_at,
                text: `@${c.user.login}: ${c.body}`,
              })),
              ...reviews.filter(r => r.body).map(r => ({
                date: r.submitted_at,
                text: `@${r.user.login} (review ${r.state}): ${r.body}`,
              })),
              ...inline.map(c => ({
                date: c.created_at,
                text: `@${c.user.login} (on ${c.path}): ${c.body}`,
              })),
            ];
            all.sort((a, b) => new Date(a.date) - new Date(b.date));
            return all.map(e => e.text).join('\n---\n');

      # secrets do not travel with the yml, communicate loudly on missing keys
      - name: Fail fast on missing OPENAI_API_KEY
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          if [ -z "${OPENAI_API_KEY}" ]; then
            echo "::error::OPENAI_API_KEY secret is not set for this repository."
            exit 1
          fi

      # retry wrappers cannot wrap `uses:` steps, so retry manually: transient
      # failures (a 401 from a fresh runner token, a hiccup reaching OpenAI)
      # should not turn the whole run red
      - name: Run Codex (attempt 1)
        id: codex_try1
        uses: ./.github/actions/codex-pr-review
        continue-on-error: true
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          conversation: ${{ steps.conversation.outputs.result }}

      # give transient failures a moment to clear before retrying
      - name: Wait before Codex retry
        if: steps.codex_try1.outcome == 'failure'
        run: sleep 20

      - name: Run Codex (attempt 2)
        id: codex_try2
        if: steps.codex_try1.outcome == 'failure'
        uses: ./.github/actions/codex-pr-review
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          conversation: ${{ steps.conversation.outputs.result }}

      - name: Post the review and publish a status
        uses: actions/github-script@v9
        env:
          REVIEW: >-
            ${{ (steps.codex_try1.outcome == 'success' &&
                 steps.codex_try1.outputs['final-message']) ||
                steps.codex_try2.outputs['final-message'] }}
        with:
          script: |
            const MARKER = '';
            const opts = { owner: context.repo.owner, repo: context.repo.repo };
            const prNumber = context.payload.pull_request.number;
            const raw = process.env.REVIEW;
            if (!raw) return;
            const body = `${MARKER}\n${raw}`;

            // one canonical comment, edited in place on every push
            const comments = await github.paginate(
              github.rest.issues.listComments, { ...opts, issue_number: prNumber });
            const existing = comments.find(c =>
              c.user?.login === 'github-actions[bot]' && c.body?.includes(MARKER));
            const posted = existing
              ? await github.rest.issues.updateComment(
                  { ...opts, comment_id: existing.id, body })
              : await github.rest.issues.createComment(
                  { ...opts, issue_number: prNumber, body });

            // go/no-go: a commit status that stays red while findings are open
            // (a missing trailer reads as "review posted", never as zero findings)
            const m = raw.match(/`, where N counts
              the unresolved rows and M how many of those are severity High.
              Always include it, even when N is 0: CI parses it into the
              status check.

          Pull request title and body:
          ----
          ${{ github.event.pull_request.title }}
          ${{ github.event.pull_request.body }}

          Conversation history:
          ----
          ${{ inputs.conversation }}

Three design choices carry the weight here.

• The review is stateful. The Feedback Summary table tracks every finding ever raised on the PR, resolved or not, across pushes. Combined with the edit-in-place comment from the workflow, a PR carries exactly one review artifact that reflects the entire lifecycle, instead of five stale comments that each reflect an update in the review process. Anyone opening the PR, human or agent, sees the full history at a glance.

• Declines are honored. Because Codex reads the whole conversation, the author can reject a finding in writing, and the rejection sticks across review cycles. This is the mechanism that makes an imperfect reviewer workable: false positives cost one written reply instead of an endless nag loop, and the objection stays on record for the next reader.

• The verdict is machine-readable. The trailing line gets parsed by the workflow into a Codex review commit status. The result is two independent signals on every PR: the workflow check tells you the review ran, and the status tells you whether anything is still open. Red status means unresolved findings; green means everything is fixed or explicitly declined. I deliberately keep the status non-blocking, since it only functions to inform. The decision to merge or not remains open, and remains human.

Note how this contract asks for more than typo hunting. Codex reviews the diff against the PR’s stated intent, ranks every finding by severity, and attaches a concrete proposed fix, the kind of review that regularly surfaces architecture-level issues no linter would ever produce.

Insight: structure beats raw model quality in a reviewer. A lifecycle table, a decline rule, and a machine-readable verdict turn free-form LLM opinions into a merge gate with a mechanism for written pushback that makes false positives cheap.

---

Close the loop: the PR turns green before you even look

The reviewer is only half the story when you look at the bigger engineering picture. The other half is teaching the author agent to respond to it, so the review round-trip does not become your job (remember: always try to remove the bottleneck). In my repos I use three Claude Code skills to support me, written as plain markdown instruction files under .claude/skills/:

• /pr-open creates the PR: analyzes the branch’s commits, runs validation locally, writes an honest title and body, and targets the right base. The PR body matters more than it seems, since the reviewer reads it as the statement of intent it reviews against. I have it spell out the full feature request and reference the Linear ticket it implements, so both the reviewer and any human arriving later get the complete picture without leaving the PR.
• /pr-iterate handles one review round: sweeps all three feedback surfaces, then triages every item into a fix, a decline with reasoning, or a defer. Fixes are implemented and validated locally; declines become PR comments explaining why. The reviewer is advisory, and “good enough to merge” is a valid verdict once the PR meets its stated scope.
• /pr-babysit is the watchdog around it all: watch the CI, fix red checks, run an iterate round when new feedback lands, and loop until every check is green and every finding is fixed or declined. Its terminal state is “merge-ready”, and it stops to report rather than ping-pong commits when the same check fails twice on the same root cause.

One ordering rule that you and your agents should always keep in mind: reply first, push after. The review kicks off on every push, and it works with the context that exists at that exact moment. Anything not yet written down (a decline, a fix summary, a comment of your own) does not exist for that review round, and the reviewer will happily re-raise every item it cannot see a response to.

The end state is a tidy division of labor. Claude writes and defends, Codex challenges and tracks, the loop runs until the PR is green with a clean feedback table. My queue contains only PRs that are already merge-ready, each carrying a review trail I can audit in one scroll: what was found, what was fixed, and what was declined and why.

Insight: automate both sides of the review. When the author agent must fix or contest every finding in writing before it can push, the human queue only ever contains merge-ready PRs with an auditable review trail.

---

What a round-trip looks like in practice

To make the mechanism concrete, here is one exchange from a backend PR of mine. The PR fixed a capacity grid that wrote more cells than a hardcoded batch cap allowed. The first review round came back with two findings:

## Feedback Summary

| Issue                           | Severity | Summary                                  | Resolved |
| ------------------------------- | -------- | ---------------------------------------- | -------- |
| Truncation beyond the batch cap | High     | Cells past the cap are silently dropped. | ❌       |
| Cap duplicates a platform limit | Medium   | Move the cap into app config.            | ❌       |

The Codex review status turned red. Claude implemented the first finding for real, then declined the second one in a PR comment (real comment, anonymized):

Addressed the High finding (hard-coded row cap in the data layer, causing avoidable 502s).

_fetch_capacity_grid now adapts instead of failing: on a short page (which, because the grid is dense at exactly days rows/person, unambiguously means the deployed cap truncated the batch) it halves batch_size and retries the same slice, shrinking until pages fit. A short page only becomes a loud 502 once batch_size == 1 [...]

On the secondary suggestion (move the cap to config): intentionally not done. The real cap lives in the database’s own hosted API settings, not an app env var. Mirroring it into app config would add a second source of truth that could silently drift from the database. Adaptive batching removes the need to know the exact value at all, which is the stronger fix. Existing capacity-grid DB test still passes; lint and type-check green.

Reply first, push after, and have Codex re-review:

## Feedback Summary

| Issue                           | Severity | Summary                          | Resolved |
| ------------------------------- | -------- | -------------------------------- | -------- |
| Truncation beyond the batch cap | High     | Fixed via adaptive batching.     | ✅       |
| Cap duplicates a platform limit | Medium   | Declined: duplicate would drift. | ✅       |

Both rows resolved, one by a fix and one by a written decline that the reviewer accepted and recorded. That second row is the entire mechanism in one line: no churn was implemented just to silence a comment, and the reasoning is now part of the PR’s history for whoever reads it next.

---

The learnings ledger: what it costs and where it bites

This setup did not come together in one go. Most of its details grew out of surprises and dead ends along the way, so I am writing the learnings down here, in rough order of money saved, to spare you from driving into the same walls:

1. Pin the model. The model input of codex-action is optional, and leaving it empty means “whatever Codex currently defaults to”. I learned this on my bill when the default silently moved to a newer, pricier model (gpt-5.5). Pins solve this problem, but need maintenance in return; currently I use gpt-5.3-codex, which is already marked deprecated on the Codex models page at the time of writing (while remaining available on the API). I revisit the pin deliberately instead of letting the default decide for me.
2. Cap the effort at medium. Higher reasoning effort catches marginally more issues for significantly more money and time. Speed matters more than it looks: the review sits inside the agent’s iterate loop, so its latency multiplies across every round of every PR, and your own waiting time is part of the price. Medium has been the issues-per-dollar and issues-per-minute sweet spot. Unfortunately, no effort level catches everything; that is what the merge-owning human is for.
3. Retry the quick failures. A fresh runner token occasionally 401s, and the first connection to the API occasionally hiccups. Two attempts with a gated fallback keep those from turning the workflow red “for nothing”, which matters once agents (and your own trust) treat red as a real signal.
4. Cancel superseded reviews. The concurrency block means only the latest commit of a PR gets reviewed. Without it, an agent pushing three fixes in a row buys you three reviews, two of them for code that no longer exists.
5. Review against the base that will receive the merge. My PRs target dev, so the review diffs against dev, and I keep the feature branch synced with that base. A stale branch gets diffed against a base that has moved on, and the review then reports phantom findings for changes the PR never made.
6. Feature branches are the prerequisite. All of this hangs off pull_request events. Direct pushes to dev or main bypass the reviewer, the status, and the audit trail entirely. The one-branch-per-change flow is what makes every change reviewable in the first place.
7. Assume PR text is untrusted input. Everything the reviewer reads (the PR body, the conversation, the code itself) is a prompt-injection surface, and the agent executes on your runner with your API key. The least-privilege token caps the blast radius on the GitHub side, and codex-action ships sandboxing and safety-strategy knobs for the runner side. Also know that GitHub does not expose secrets to workflows triggered from forked PRs, so this pattern assumes same-repo branches from trusted contributors. Review the trust model before copying it into an open-source repo.

---

You are the editor-in-chief now

The point of this pipeline was never to remove the human from the loop, its intent is to uplevel you. I stopped being the first reader of every diff and became the person who reads reviews, adjudicates the occasional standoff between two models, and owns the standards written into the review prompt. When Claude and Codex agree, the PR is probably fine. When they disagree, that disagreement is a precision-guided pointer at exactly the code that deserves my attention. Where Claude sometimes acts sloppy, Codex tends to over-engineer, holding that balance in your own hands is a great place to be.

Reviewing the reviews (so meta) is a better job than proofreading agent output at agent speed, and it degrades gracefully. On a lazy day I merge green PRs on the strength of the trail, on a careful day I read the feedback table and spot-check the declines. Either way, nothing reaches the merge button on one model’s confident say-so.

Key insights from this post

1. Hallucinations are a constant, throughput is what changed. Any process that requires a human to read every line caps your agents at human reading speed, so the very first review pass has to be automated.
2. An author’s bugs are by definition plausible to the author, and research confirms LLMs are positively biased toward their own output. Review value comes from decorrelation, by using a model from a different provider that fails differently, and different is what catches what you miss.
3. The model call is the easy part. Reviewer reliability comes from ordinary CI engineering around it.
4. Structure beats raw model quality in a reviewer. A lifecycle table, a decline rule, and a machine-readable verdict turn free-form LLM opinions into a merge gate with a mechanism for written pushback that makes false positives cheap.
5. Automate both sides of the review. When the author agent must fix or contest every finding in writing before it can push, the human queue only ever contains merge-ready PRs with an auditable review trail.

Final insight: work multimodel. Your agents will sometimes hand you confident fiction, and models from different providers fill each other’s gaps, so you don’t have to. A second-provider reviewer with memory and a contract, sitting between your agents and your main branch is a great place to start, but it is not the only one. Whenever one model’s output is about to matter, a second opinion from a different lab is the cheapest insurance you can buy.

---

Follow me on LinkedIn for bite-sized AI insights, Towards Data Science for early access to new posts, or Medium for the full archive.