SchAI Benchmark
Electronic design agent evaluation
Introduction
SchAI Benchmark evaluates whether electronic design agents can turn requirements or references into editable engineering artifacts. It examines the complete process: understanding inputs, using design tools, organizing components and nets, and producing inspectable projects. Fluent explanations and tidy drawings are only part of the assessment; device definitions, parameters and connections must also meet task requirements.
Tasks are grouped by their main objective. Schematic generation examines open-ended design and module coordination. Reconstruction examines interpretation of references, parameters and topology. Symbol creation examines the conversion of device references into pin definitions and package mappings. Diagnosis examines issue localization, explanations and false positives. Each category is evaluated separately.
Difficulty is assigned before execution, using circuit scale, module count, dependencies, constraint complexity and reasoning depth. Inputs, execution records, artifacts and checks are retained. Finishing a run is distinct from passing acceptance: every required condition for that task version must be met. Difficulty does not change with a run’s outcome and is not an additional score weight.
Results describe artifact quality and execution efficiency. Quality scores are current human overall assessments, not measurements calculated from a uniform quantitative rubric or substitutes for per-case acceptance. Duration, model calls and tokens describe execution cost; cache hit rate describes context reuse. Fewer calls or more caching do not demonstrate greater circuit correctness.
Task categories and difficulty
The four categories have different inputs, outputs and acceptance criteria. Symbols created as part of a schematic task remain within that task and are not counted as separate symbol-creation cases. Diagnosis assesses identification and explanation, not automatic repair.
| Task category | Why evaluate it? | What we examine |
|---|---|---|
| Schematic generation | Turn requirements into complete circuits | Functional completeness, component selection and module connections |
| Circuit reconstruction | Interpret reference circuits accurately | Device parameters, polarity and net relationships |
| Symbol creation | Convert device references into usable symbols | Pin completeness, numbers, names and package mappings |
| Circuit diagnosis | Find and explain existing issues | Fault localization, causes and false positives on correct circuits |
Difficulty progresses from local interpretation and execution (Easy), through connected modules (Medium), to system reasoning and multiple constraints (Hard). It is determined from the inputs and acceptance requirements before execution, not from success or failure.
| Task category | Easy | Medium | Hard |
|---|---|---|---|
| Schematic generation | One module, clear requirements | Power and interface coordination | Full system, multiple constraints |
| Circuit reconstruction | Clear image, one module | Dense layout, cross-region nets | Large, multi-sheet reconstruction |
| Symbol creation | Few pins, direct mappings | Many pins, multiplexed functions | Complex packages, multiple units |
| Circuit diagnosis | One clear local issue | Several possible causes | Coupled faults across modules |
Results
Each task category is evaluated separately, showing current artifact quality and the mean time required to complete the task.
| Task category | Assessment / 100 | Mean duration | Mean model calls | Cache hit rate | Mean input tokens | Mean output tokens |
|---|---|---|---|---|---|---|
| Schematic generation | 85 | 828 s | 25.3 | 94.1% | 1,786,336 | 22,801 |
| Circuit reconstruction | 95 | 136 s | 8.7 | 87.8% | 197,053 | 3,792 |
| Symbol creation | 100 | 87 s | 7.0 | 84.5% | 197,342 | 2,440 |
| Circuit diagnosis | 95 | 27 s | 4.3 | 88.8% | 42,655 | 479 |
Loop rounds count model calls per task. Token counts are accumulated across calls; input tokens include cached input. Cache hit rate is the total cached input tokens divided by total input tokens.
Scores are human assessments of current artifacts, not case pass rates. Mean durations come from recorded runs. Results reflect project artifacts and checks, not physical hardware validation.
Conclusion
SchAI can produce editable schematics spanning multiple modules, but open-ended design remains a primary area for improvement. It requires coordinating functions, components, parameters and constraints; completeness and consistency still need engineering review. Higher call counts and token usage describe execution overhead, not evidence of design correctness.
Tasks with explicit references currently produce more mature artifacts. Typical application circuits and pin references provide a basis for checking devices, parameters, connections and package mappings. Reconstruction scores 95 and symbol creation scores 100 in this assessment. These results support reference reconstruction and symbol creation as useful engineering assistance; 100 does not imply perfect accuracy for every device, package or industrial use.
Diagnosis can locate common connection, polarity and parameter issues and explain possible effects. Finding known faults is different from excluding all possible problems. Missed faults, false positives and root causes involving multiple issues still require further evaluation. Current results cover diagnosis, not automatic repair or full engineering and hardware validation.
SchAI currently serves as an engineering assistant for reference organization, circuit reconstruction, symbol creation and preliminary diagnosis, and provides editable starting points for open-ended designs. Further evaluation should add per-case acceptance and repeated runs to examine correctness, consistency and execution efficiency. Capability claims should develop with the evidence.