We evaluated 12 harnesses on Runta across 30 challenging agentic tasks, all using the same Kimi K3 model.

CodexDSH CreatorClaude CodePiDSH PTCDSH StandardOh My PiKimi CodeDSH MinimalExo HarnessOpenCodeHermes
Pass rate
50.0%
55.6%
61.1%
66.7%
$1.0
$2.0
$5.0
$10
$20
Median cost per pass (failures included)
View resultsGitHub

Pass rate

Median cost per successful task

Median cost per pass (failures included)

Median cache hit rate per successful task

Median time per successful task

Beyond the numbers

  1. 01
    OpenCode: failures excluded.

    It only covers 15 passes. Count failed attempts and the number becomes $3.24 per pass.

  2. 02
    Cache hit rate is not cost.

    A cached 300-turn failure can still burn more than a short cache miss.

  3. 03
    Quality and cost can diverge.

    Claude Code passes 19 tasks, but reaches $18.34 in effective cost per pass.

Runta

Build and run your agents on Runta.

Tested harnesses

Codex
v0.148.0
DeepSeek Harness
v0.1.0-rc.8
Claude Code
v2.1.237
Pi
v0.84.2
Oh My Pi
v17.4.0
Kimi Code
v0.37.2
Exo Harness
v0.1.0
OpenCode
v1.18.19
Hermes
v0.20.4
  • Evaluated on Runta agent runtimes. All harnesses and the task environment are prepared once as a golden checkpoint. Every run is a fresh restore with identical vCPU, memory, disk size, disk contents, and memory state.
  • Kimi K3 is served by Fireworks.