Coduck logo Coduck

Head to head

One 100-file pull request. Three reviewers.

Same commit, same diff. Every finding below is what each tool actually reported.

100files changed
7defects in the change
3reviewers

Coduck

0findings

Six real defects, including the two that change what the stored data means.

Greptile

0findings

Four, one of which Coduck missed. A real second opinion.

CodeRabbit

0finding

One, in its summary. Its two failed checks were the PR title and docstring coverage.


Every finding, from all three

Seven defects were in the change.

Row four is a defect Greptile found and Coduck did not.

Finding
Coduck
Greptile
CodeRabbit
A timeout= the client does not acceptEvery collector raises TypeError on its first call.
must fix
P1
in summary
Collectors never reachedNo registry entry, so none of it runs.
should fix
P2
 
quotes_expiry unusedThe routes it serves do not exist.
nit
P2
 
External JSON parsed unvalidatedInto dict[str, Any], no schema.
 
P2
 
None conflates three different failuresAll three are written down as found: false.
must fix
 
 
Domain not URL-escapedThen stored as evidence.
nit
 
 
Untyped record against the normalised schemaUnpicking it means editing 100 files.
should fix
~partly
 

Two of the seven

The floor, and the ceiling.

The one every tool caught, and the one only Coduck did.

All three found this

collectors/attestation.py
18+ resp = client.get(url, timeout=30)
coduckMust fix

PoliteClient.get takes no timeout. Every collector raises TypeError on its first call.

Only Coduck found this

collectors/breach.py
42 result = client.get(url)
43 if result is None:
44+ return {"found": False}
coduckMust fix

A robots refusal, a timeout and a 500 all arrive as None. All three are filed as the company having no breaches.

The second is a claim about what the stored data means, not about the code, and making it meant reading the docstring of a function the diff never touched.


What this does not show

One pull request, run by us.

It is a single change

One pull request, one codebase. A different change could rank these differently.

We ran it

We chose the pull request and judged the findings. Nobody independent has checked it.

Coduck missed one

Greptile found the unvalidated JSON. Coduck did not.

The tools do different jobs

CodeRabbit’s checks are a policy layer. A team that wants docstring coverage enforced is served by them.

Run it on your own pull request.

The CLI is free, local, and takes one command. Compare it yourself.

Read the docs