OpenHands: An Open Platform for AI Software Developers as Generalist Agents

Reading guide

Problem and motivation

Software development is among the most expressive interfaces through which humans interact with the digital world. While large language models (LLMs) can generate isolated snippets, real-world software engineering demands long-horizon interaction: navigating repositories, modifying multi-file architectures, executing bash commands, reading documentation via web browsers, and self-debugging using execution feedback.

Prior agent frameworks often suffered from key architectural gaps: lack of isolated execution environments, stateless code runners, missing multi-modal web browsing capabilities, or absence of standardized evaluation tooling across heterogeneous domains. OpenHands (formerly OpenDevin) addresses this need with an open-source platform (MIT licensed) offering sandboxed runtime environments, extensible Agent-Computer Interfaces (ACI), multi-agent coordination, and standardized benchmarking.

Key ideas

• Event Stream Architecture: User instructions, agent actions, and runtime observations are tracked chronologically in an event stream that unifies communication across frontends, agents, and sandboxed runtimes.
• Programming Language (PL) Primitives: Instead of brittle custom APIs, agent interactions are grounded in general actions: IPythonRunCellAction, CmdRunAction, and BrowseInteractiveAction (via BrowserGym).
• Extensible AgentSkills: A modular Python package preloaded into the IPython kernel provides high-level utilities (e.g., AST-grounded line editing edit_file, PDF parsing, multimodal image extraction) without reinventing existing packages.
• Agent Delegation: Specialist agents handle domain-specific workflows (e.g., delegating web tasks via AgentDelegateAction to a dedicated BrowsingAgent).
• Quality Control via Mocked Testing: Routine continuous integration tests prompt regressions, actions, and sandbox stability using deterministic prompt-response caching instead of expensive real-time API calls.

Method (with the key equations)

The OpenHands execution cycle operates on a discrete step function where the agent maps an environment state $\mathcal{S}$ to an action $a\in \mathcal{A}$: $$a_t=\text{Agent.step}({\mathcal{S}}_t)$$ Here, the state encapsulates the chronological history of prior actions and observations: $${\mathcal{S}}_t=\{(a_0,o_0),(a_1,o_1),\dots ,(a_{t-1},o_{t-1})\}\cup {\mathcal{M}}_t$$ where ${\mathcal{M}}_t$ tracks auxiliary state metadata (cumulative LLM cost, active multi-agent delegation stack, configuration parameters).

The action space $\mathcal{A}$ includes:

• CmdRunAction(command): Runs arbitrary bash commands in the target Linux shell.
• IPythonRunCellAction(code): Executes stateful Python code in a persistent Jupyter kernel, exposing AgentSkills.
• BrowseInteractiveAction(code): Executes Playwright-driven BrowserGym primitives (click(bid), fill(bid, val), goto(url), etc.).
• AgentDelegateAction(subtask, agent_name): Pushes a sub-goal onto the agent stack to summon a specialist.
• AgentFinishAction(): Concludes execution.

Execution occurs within a secure client-server Docker sandbox. Base Docker containers are extended with an OpenHands REST API server (runtime.py), tagged using a dual-tagging mechanism: a content MD5 hash (target_image_repo:target_image_hash_tag) for strict reproducibility, and a semantic generic tag (runtime:oh_v{VERSION}_{BASE_IMAGE}_tag_{IMAGE_TAG}) for cached base-layer builds.

Results (with the most important numbers)

OpenHands evaluates generalist agents across 15 benchmarks spanning software engineering, web browsing, and general assistance without per-benchmark prompt tailoring:

• SWE-Bench Lite (300 instances, w/o hints): CodeActAgent v1.8 with claude-3-5-sonnet@20240620 achieves a $26.0\%$ resolve rate (average cost $1.10 per instance), outperforming SWE-Agent ($18.0\%$) and AutoCodeRover ($19.0\%$), and approaching Aider ($26.3\%$).
• HumanEvalFix (164 instances): CodeActAgent v1.5 with gpt-4o-2024-05-13 solves $79.3\%$ in a 0-shot self-debugging setup (compared to StarCoder2-15B's $48.6\%$).
• WebArena (812 instances): BrowsingAgent v1.0 achieves $15.5\%$ using claude-3-5-sonnet and $14.8\%$ using gpt-4o, while CodeActAgent v1.8 via delegation reaches $15.3\%$.
• MiniWoB++ (125 environments): Zero-shot BrowsingAgent v1.0 with gpt-4o achieves $40.8\%$.
• General Assistance & Reasoning:
  • GPQA Diamond: CodeActAgent v1.8 + claude-3-5-sonnet scores $52.0\%$ (surpassing expert human baseline at $81.2\%$, but outperforming few-shot CoT GPT-4 at $38.8\%$).
  • GAIA (L1 Validation): GPTSwarm v1.0 with gpt-4o reaches $32.1\%$ ($13.2\%$ for AutoGPT).
  • AgentBench (OS subset): CodeActAgent v1.5 achieves $57.6\%$.
  • MINT (Math / Code): CodeActAgent v1.5 hits $77.3\%$ on math and $50.0\%$ on code.

Limitations and open questions

• Long File Edits: The current editing primitives struggle when handling large codebases or long source files where line-number shifts introduce edit conflicts.
• Multimodal Integration: Although helper skills (parse_image, parse_pdf) extract text or image summaries via external vision models, native browser and terminal multimodal reasoning is not yet fully unified.
• Workflow Automation: Workflows still rely heavily on hand-crafted system prompts and agent orchestration loops rather than automatically optimized graph architectures (e.g., GPTSwarm).