CodeActionBench
Evaluating Agentic Code-as-Policy for Embodied Manipulation
Can general-purpose models turn visual understanding into robot action? We test the code they write, execute, and revise across 25 manipulation tasks.
Leaderboard
Verifier success rate (%)
25 tasks × 9 configurations × 3 attempts · 675 episodes
- GPT-6 Astra (Codex)?GPT-6 Astra (high) runs in the Codex agent, using the same robot tools over MCP.73.33
- Claude Opus 549.33
- Claude Code (Opus 5)?Default rows use the shared Reference harness. This row uses Claude Code 2.1.212 with vendor-mcp-direct; tasks, D0 tools, budgets and verifier stay fixed, so the complete agent stack is being compared.45.33
- Gemini 3.6 Flash20.0
- GPT-5.616.0
- Qwen 3.8 Max14.67
- Grok 4.612.0
- Kimi K3 Intl12.0
- Claude Sonnet 52.67
Evidence views
Demos
Recorded execution, paired with agent-visible observations. Open a demo for the full trajectory.
Sort byAverage Success Rate (ASR): verifier successes divided by scoreable episodes, including the laptop and dual-shoe regrade. Scoreable failures count as zero; pending and INFRA episodes are excluded.Number of the 25 tasks with at least one successful repeat.Number of the 25 tasks successfully completed in all three repeats.Median episode wall-clock time across the agent's 75 accepted attempts. It includes provider latency and is descriptive, not part of rank.Median charged outer tool calls across the agent's 75 accepted attempts. Internal primitives inside run_code are not charged again.
| # | Agent | Average Success Rate (ASR): verifier successes divided by scoreable episodes, including the laptop and dual-shoe regrade. Scoreable failures count as zero; pending and INFRA episodes are excluded. | Number of the 25 tasks with at least one successful repeat. | Number of the 25 tasks successfully completed in all three repeats. | Median episode wall-clock time across the agent's 75 accepted attempts. It includes provider latency and is descriptive, not part of rank. | Median charged outer tool calls across the agent's 75 accepted attempts. Internal primitives inside run_code are not charged again. | EvidenceOpen a published per-turn trajectory or compare model trajectories in the Arena. |
|---|---|---|---|---|---|---|---|
| 01 | GPT-6 Astra (Codex)?GPT-6 Astra (high) runs in the Codex agent, using the same robot tools over MCP. | 73.3355/75 | 22/25 | 15/25 | 8.0 min | 30 | |
| 02 | Claude Opus 5 | 49.3337/75 | 19/25 | 7/25 | 19.9 min | 34 | |
| 03 | Claude Code (Opus 5)?Default rows use the shared Reference harness. This row uses Claude Code 2.1.212 with vendor-mcp-direct; tasks, D0 tools, budgets and verifier stay fixed, so the complete agent stack is being compared. | 45.3334/75 | 14/25 | 9/25 | 22.7 min | 36 | |
| 04 | Gemini 3.6 Flash | 20.015/75 | 8/25 | 3/25 | 13.0 min | 42 | |
| 05 | GPT-5.6 | 16.012/75 | 9/25 | 1/25 | 10.6 min | 56 | |
| 06 | Qwen 3.8 Max | 14.6711/75 | 6/25 | 2/25 | 32.5 min | 36 | |
| 07 | Grok 4.6 | 12.09/75 | 6/25 | 1/25 | 15.9 min | 56 | |
| 07 | Kimi K3 Intl | 12.09/75 | 5/25 | 0/25 | 45.5 min | 55 | |
| 09 | Claude Sonnet 5 | 2.672/75 | 2/25 | 0/25 | 17.6 min | 56 |
Primary metric · verifier successes / scoreable episodes25 tasks · 3 episodes per task · every task weighted equallyTime and charged calls are descriptive resource metrics and never alter rank
Submit your results
Leaderboard submissions are reviewed against the frozen public release identity before publication.
- Run all declared model-task episodes with the frozen public task set, tool surface, seed and budget.
- Bundle the run identity, verifier results, trajectory manifest and media manifest; include a SHA-256 for the archive.
- Open a GitHub issue with the model and system name, paper or report link, archive location and checksum.
Agent arena
Head-to-head verifier comparison
Compare two agents on the same tasks, with outcomes and complete trajectories.
Model Acodex-astra
VSModel Bclaude-opus-5
17Both covered
5A only
2B only
1Neither covered
Benchmark interface
How agents see, code, and act
Explore the interfaceClose+
Benchmark interface
How agents see, code, and act
Task instructionLift the pot
Evaluated agentMultimodal LLM
Reference harness / Codex CLI / Claude Code
Policy code
Recorded excerptAstra · lift_pot · turn 6
pair2=capture_motion_pair(arm='right',dx=-0.07,dy=0,dz=0)Tool calls→←Results / feedback
Robot API
Execute
Physical interactionOutside the interface×Depth sensing×Perception / grasp specialists×Privileged scene state×Pre-built skills
The agent decides what to observe, estimate, and execute.
What does the agent need to establish?Browse by purpose, not execution order
Choose a geometric method, its evidence, and its assumptions to obtain a ray, depth, or 3D point.
Where a specialist or simulator supplies spatial answers, the agent supplies pixels, correspondences, planes, or size priors. The API performs the requested calculation.
Optional physical measurement · probe_contact_along
Contact probing is a separate source of physical evidence. It moves the robot and may change the scene; it is not historical visual memory.
Resources
Results, paper & code
@misc{lyu2026codeactionbench,
title = {CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation},
author = {Yiheng Lyu},
year = {2026},
note = {Release website preview}
}