Benchmarking LLMs on VLSI physical design: foundational knowledge, root-cause analysis and report reading, through to exact place-and-route, formal-verification and ECO scripts.
Click a model to add it to the capability profile.
0255075100
Score matrix
Every model on every category (%). Click a column header to sort by it. Hatched cells: the model is text-only and these tasks need image input.
0%100%† overall averaged over the 6 text-only categories
Capability profile
Compare up to three models across all eight categories. Understanding categories run clockwise from the top; script-generation categories fill the lower left.
Models
Categories
343 tasks in eight categories. Click a card to rank models on that category.
PlaceReasoner-Bench: end-to-end macro placement
16 tasks: 8 open-source designs, each at a square (1:1) and an elongated (2:1) die. A method outputs every macro's position and orientation, and the full open flow (Yosys, OpenROAD, Nangate45, FakeRAM 2.0) then places, routes and signs off the design. Scores are post-route PPA. Our agent, PlaceReasoner-Beta, runs on Claude Opus 4.8; the same agent on two open-weight Qwen3-VL backbones is shown for comparison. The benchmark is part of the open-source release.
Post-route TNS per task
Detailed results
Simplified from Tables 6 and 7 of the PlaceReasoner-Beta paper, without the CWI / CWS ablation rows. Bold: best WNS / TNS on the task among runs with fewer than 1,000 DRC violations; greyed rows exceed that budget. Runtime: macro-placement time for the classical tools, total model-driver time for the agents. Tokens: model input + output.
Placements produced by PlaceReasoner-Beta
Fig. 9 of the paper. Macro placements produced by PlaceReasoner-Beta (Claude Opus 4.8) on 15 PlaceReasoner-Bench tasks, with each macro's pin side marked: (a)–(h) the eight designs at 1:1 and (i)–(o) seven of them at 2:1 (bp_fe at 2:1 is shown separately in the paper). Macros are grouped against the periphery and each group's pin side faces the standard-cell region it serves, leaving that region contiguous.
Method
All models are called as plain base-model APIs: no tools, no retrieval, no agent loop, one stateless turn per task.
Scoring
Judged (Foundational, Root-Cause, Report): rubric score 0–10 from an LLM judge, shown as %.
pass@1: one sample. pass@5: five separate samples; any-correct for rule-checked tasks, mean score for judged tasks.
Overall: unweighted mean of the category scores, so every category counts equally regardless of size.
Reading the results
LATEST models were run on 2026-09-23 (reasoning effort medium, 32K output cap) and judged blind by GPT-5.4-mini.
EARLIER models come from the paper's earlier evaluation, with a different judge for the three judged categories. Compare those categories within a round. Rule-checked categories use identical checkers in both rounds.
STA, ECO and Report have only 10–11 tasks each, so one task moves the score by about 9–10 points.
An open-source version runs on open tools only and contains 133 QA tasks plus PlaceReasoner-Bench.
Citation
@article{li2026pdagentbench,
title = {PDAGENT-BENCH: Characterizing, Grounding, and Architecting LLM/VLM Agents for VLSI Physical Design},
author = {Li, Qiufeng and Chen, Rongqian and Cheng, Quan and Wang, Chengxuan and Tang, Sizhe and Ho, Chia-Tung and Pan, David Z. and Lan, Tian and Cao, Weidong},
journal = {arXiv preprint arXiv:2606.17253},
year = {2026}
}
