Benchmarks

Measured on real pull requests, from two open source codebases.

Fifteen merged pull requests from cal.com and PostHog, reviewed by Coduck with no knowledge of what the human reviewers had said. Every number below comes from that run. Where we have not measured something, the page says so.

Precision

82%of 79 findings

Each finding Coduck raised was checked against the merged diff. 82% described a real problem in the code.

held up did not

Recall · cal.com

77%of 32 goldens

Of the issues cal.com’s own reviewers raised on those pull requests, Coduck independently found 25 of 32.

found missed

Two things we will not dress up. These figures are self-adjudicated: we judged our own findings against the merged diff, and no third party has audited that. And three of cal.com’s 35 golden comments turned out to be examples reproduced in public documentation, which a model may well have seen before. Coduck found all three. They are excluded from the 32 above, because a number that counts them is measuring memory rather than review.

What was reviewed

Two large TypeScript codebases with busy review cultures, chosen because their pull requests carry substantive reviewer comments that can serve as an answer key.

calcom/cal.com

pull requests

golden comments

Coduck findings

PostHog/posthog

pull requests

golden comments

Coduck findings

What it reported

79 findings across the 15 pull requests, sorted by how hard Coduck pushed. Only the first two groups would fail a check; nits are reported and never block.

17 must fix

41 should fix

21 nits

Must fix, 17 Should fix, 41 Nit, 21

The answer key

cal.com’s 35 golden comments, graded by the severity their own reviewers assigned. More than half are High or Critical, so this is not a set made of style nits.

Critical2

High15

Medium10

Low8

Every pull request in the run

Nothing was dropped for looking bad. PR 14740 returned no findings at all and is counted against us in the recall figure.

Pull request	Golden	Coduck findings
cal.com #10600	5	8
cal.com #10967	6	12
cal.com #11059	9	14
cal.com #14740	6	0
cal.com #22532	4	11
cal.com #7232	3	5
cal.com #8087	2	3
PostHog #29131	1	7
PostHog #27601	1	5
PostHog #26649	1	4
PostHog #25957	1	1
PostHog #22766	1	1
PostHog #20945	1	2
PostHog #20394	1	4
PostHog #18781	1	2
Total	43	79

How it was run

Reviewed at the merge base

Each pull request was reviewed against git merge-base origin/main pr<n>, so Coduck saw exactly the diff the human reviewers saw, and nothing that landed afterwards.

Three reviewers

--reviewers architecture,security,correctness. The product reviewer was not run, because these pull requests have no ticket to judge against.

Goldens chosen before the run

cal.com goldens are a curated set with severity and category assigned per comment. PostHog goldens are the raw reviewer comments, unlabelled. Neither was edited after seeing what Coduck reported.

Judged against the merged diff

A finding counts only if the code it describes actually behaves that way in the merged pull request. Wording does not have to match a golden comment for a match to count.

The exact command, for anyone who wants to repeat it:

# for each pull request in the set
b=$(git merge-base origin/main pr$pr)
h=$(git rev-parse pr$pr)

coduck review --target $h --base $b \
  --reviewers architecture,security,correctness