Introducing CADGenBench: measure how well AI systems produce engineering-grade 3D parts!

While current models can generate 3D parts, they are far from precise enough to build functional parts. We built a benchmark to systematically measure their capabilities on two tasks:

1. Generation from an engineering drawing of a part 2. Editing: given an existing STEP file and a requested change

The benchmark is tool-agnostic. It makes no assumptions about how you build the model. You can vary the LLM, and you can vary the environment. Use build123d, Onshape, Autodesk, or a model without an LLM entirely. We open sourced the scoring engine and a reference baseline on top of build123d.

A collaboration between Hugging Face and @mecadoinc!

Submission space: https://huggingface.co/spaces/HuggingAI4Engineering/CADGenBench Code repository: https://github.com/huggingface/cadgenbench

3/ Topology match

Geometry is not the whole story.

Two parts can look similar while differing in their fundamental topology: • 4 holes instead of 2 • 2 disconnected pieces instead of 1 • a missing enclosed cavity

To measure this, we compare the Betti numbers of the generated and reference parts: • b₀ = number of connected components • b₁ = number of through-holes / handles • b₂ = number of enclosed cavities

These quantities are independent of the CAD system used to create the model.

AI is moving beyond text, images, and code.

Engineering artifacts are becoming a new class of model outputs and evaluating them requires different tools than we use for text, code, or images.

Today we're excited to release CADGenBench, a benchmark for CAD generation and editing.

- Given an engineering drawing → generate a valid 3D CAD model - Given a STEP file + change request → edit it correctly

The benchmark is tool-agnostic: any CAD stack works (Fusion, Onshape, build123d, SolidWorks, etc.). Submissions are simply STEP files.

Models are scored on: * geometric accuracy * topology correctness * interface compatibility * CAD validity

The benchmark is open, the ground truth is private, and the leaderboard is live.

Since CAD evaluation is surprisingly subtle, here's how the metrics work 🧵

0/ CAD validity

Before any comparison with the ground truth, we first check that the submitted STEP file represents a valid CAD model.

The model must be: ✓ well-formed ✓ watertight ✓ meshable ✓ manifold

If any of these checks fail, the overall score is zero.

1/ Shape similarity

The first question is whether the generated part has the correct overall geometry.

We combine two complementary measures: • Surface Distance F1 → are the surfaces located where they should be? • Volume IoU → does the part occupy the same volume?

Together, they measure agreement in both surface placement and occupied material.

2/ Interface match

Many CAD features exist to interface with other parts.

We therefore evaluate mating features such as: • bolt holes • slots • bosses • pockets

Each feature is represented as a region that should either remain empty (keep-out) or contain material (keep-in).

This captures errors that may have little impact on overall shape similarity but would prevent a part from assembling correctly.