cad-eval · CADSmith protocol
Does the kernel reproduce the drawing?
The field's CAD-generation benchmarks (SimBench, and the
LLM-CAD line generally) score a generated part by asking a
language model whether it looks right — correctness by opinion.
CADSmith (arXiv 2603.26512) showed the alternative: drive the
scoring from exact, deterministic measurements taken by the
geometry kernel itself, and a kernel-checked refinement
loop pulls mean Chamfer Distance from 28.37 mm down to 0.74 mm.
cad-eval takes that protocol and runs our own
B-rep kernel through it.
Every score is Chamfer Distance, F1 at a 1 mm threshold, and
volumetric IoU at 1 mm voxels — computed by the kernel, with a
bit-op and joule receipt attached. No fixtures, no RNG, no
judge. Below: physical-brep building each case from
primitives and booleans, sampled and Chamfer-compared against
the procedural reference.
| case | tier | geometry | surface pts | CD (mm) |
|---|---|---|---|---|
| t1_box | T1 | 12×8×6 mm box | 6,888 | 0.299 |
| t1_cube | T1 | 10 mm cube | 8,436 | 0.301 |
| t1_cylinder | T1 | r5 × 12 mm cylinder | 86,224 | 0.282 |
| t1_cylinder_squat | T1 | r8 × 4 mm disc | 157,784 | 0.272 |
| t2_washer | T2 | r8×3 disc − r4 bore (boolean) | 141,312 | 0.551 |
| mean Chamfer Distance over 5 cases | 0.341 | |||
Why a perfect kernel doesn't score zero
The honesty is in the floor. The brep samples lie on the true surface; the reference points are voxel-boundary centers on a 0.5 mm grid. The two discretizations can't coincide, so the comparison has a built-in sub-voxel floor of roughly 0.25–0.5 mm — a faithful kernel lands at that floor, not below it. Four cases sit at 0.27–0.30 mm, exactly there. The washer is the honest outlier at 0.55 mm: a boolean subtraction leaves a bore edge, and an edge is precisely where two surface discretizations diverge most. The number tells you where the geometry is hard, which is the point of measuring instead of asking.
The position
An LLM-judged CAD score is an impression that cannot be
reproduced or audited. A kernel-measured score is a number that
means a fixed thing and lands the same every run. The harness
and these numbers are in the tree
(crates/cad-eval, examples/brep_agreement);
any generator — a learned one included — is scored on the same
surface, and earns its place by the measurement, not by a
verdict.