✳ainav.

guides

How to Evaluate AI Software Engineering Agents Before You Pilot

A documentation-based guide to piloting repo or cloud coding agents: isolate the environment, start on a non-sensitive repo, and require a human to review every diff. Official Copilot docs add a second reviewer and do not auto-run Actions. Codex is described from official docs only and was not invoked.

How to Evaluate AI Software Engineering Agents Before You Pilot

A software-engineering agent is not the same product as the assistant that completes a line in your editor, and it is not the same product as a pull-request review bot. The agent you are evaluating here takes a task, works in an isolated environment, and comes back with a diff that a human still has to read. Teams that skip that last requirement, or that grant a new vendor access to a production monorepo on day one, are buying operational risk rather than speed.

This documentation-based guide was reviewed on November 10, 2026 (Asia/Shanghai). We did not invoke Codex, start a coding CLI, or run any agent against a repository. Descriptions come from official Devin, GitHub Copilot, Codex, and Jules documentation opened that day. Marketing lines about merge-ready pull requests are vendor claims, not a reason to skip review. This article does not cover exploits, vulnerability creation, or bypassing security controls.

Three exclusive coding jobs, plus a required review path

Daily IDE assistance—inline completion, chat, and in-editor agent mode—is already covered in Cursor vs GitHub Copilot vs Trae for daily coding. A review bot that comments on diffs is a third job, covered in Qodo vs GitHub Copilot vs Cursor for AI code review. Those published articles are boundaries here, not extra rows in the same table.

The core set for a cloud or repo-level coding agent is Devin, GitHub Copilot (specifically the official coding or cloud agent, which lives under that same tool ID), Codex, and Jules. GitLab Duo Agent Platform, Amazon Q Developer, and Kiro are real products, but they are not scored here because their official agent docs were not opened first-hand for this review.

A process you should insist on, regardless of vendor, is: a scoped issue, work in an isolated environment, a diff or pull request, and mandatory human review before merge. That sequence is a control, not a ranking of the three exclusive jobs above.

Diagram of three mutually exclusive coding jobs—IDE daily assistant, repo or cloud coding agent, and review bot—shown as separate columns, with a required human-review path under the agent job only.

AINav original diagram based on official docs. Sources: Devin, Devin environment, Devin secrets, GitHub: review Copilot output, GitHub Copilot risks and mitigations, Introducing Codex, Jules getting started, Jules environment. Not an official screenshot or a test result.

Devin: a cloud engineer in its own machine, still a pull request

Devin’s official product page describes an “autonomous software engineer” that can run in the cloud or on your machine, use its own browser for tests, and work until “the PR is ready to merge.” Official environment docs are more operational: Devin runs in a Linux VM, with optional Windows or macOS snapshots, after you connect source control and grant repository access. Official secrets guidance says secrets are encrypted at organization, personal, repo, or session scope; dedicated service accounts are recommended; and you should not put secrets in blueprint YAML, setup commands, or a committed .env. TOTP is described for Devin-dedicated accounts, not for personal two-factor setups.

Read the “ready to merge” line as marketing. You still review the PR. Devin also sells a separate Review product, including an Autoapprover described in official docs as able to post real approvals that may satisfy branch protection. That is a different job, and a dangerous default if it replaces a human on a sensitive repo. Enterprise marketing on the product site includes zero data retention with model providers and optional private networking; confirm those terms in a contract, not only on a homepage.

Public list prices were not taken from a first-hand official pricing table in this review and are not listed here.

GitHub Copilot’s coding agent: branch limits and a second reviewer

GitHub Copilot is already a daily IDE assistant. The feature in this evaluation is the official coding or cloud agent that takes an issue and opens a pull request. Official risks and mitigations state that only users with write access can trigger the agent; the agent is limited to the pull-request branch or a new copilot/ branch and to existing branch protections; and it cannot mark its own PRs ready, approve them, or merge them. If the repository already requires one approval, a PR opened under the app identity needs an extra approval. Official review guidance is direct: review Copilot PRs as thoroughly as any other contribution; the assigning user’s approval does not count when approvals are required—another reviewer must approve; Actions workflows do not run automatically when Copilot pushes; and you should inspect changes, especially under .github/workflows/, and use “Approve and run workflows” only after that inspection.

Those controls are the purchase value as much as the agent itself. Plan seats and GitHub AI credits from official GitHub billing pages for your org; this review does not restate directory plan rates.

Codex and Jules: official agent products, reviewed from docs only

Codex is discussed here as a documented OpenAI product. It was not invoked for this article.

OpenAI’s Introducing Codex post, dated May 16, 2025, describes a cloud software-engineering agent in an isolated environment preloaded with the repository, able to write features, answer codebase questions, fix bugs, and propose pull requests, with optional AGENTS.md guidance. The same post tells users to manually review and validate all agent-generated code before integration and execution. The page itself marks parts of that launch post outdated—including, as of June 3, 2025, internet access during tasks, and later notes on team workflows. Do not treat 2025 research-preview rollout or token language as current pricing or networking policy. Recheck OpenAI’s current Codex documentation for how cloud tasks, networking, and ChatGPT plan entitlements work now.

Jules, Google’s official coding agent, is documented as experimental. Official getting started pages say it integrates with GitHub, works in a virtual machine where it clones your code, installs dependencies, and modifies files, and that you can review and approve a plan before code changes. It looks for AGENTS.md in the repository root. Official environment docs describe a secure, short-lived Ubuntu VM, optional setup scripts, and environment snapshots for later tasks. Repo access is a grant you choose; start with a non-sensitive repository.

A pilot that starts on a throwaway repo

Use a non-sensitive repository, scoped permissions, and secrets that are not production credentials. Then check six things:

  1. Job fit. If you mainly want completions in the editor, you are in the daily-coding article. If you mainly want comments on human PRs, you are in the code-review article.
  2. Environment and access. Confirm the isolated VM or branch model, which repos the agent can see, and whether the network path is acceptable.
  3. Secrets. Follow official Devin-style rules even on other tools: dedicated service accounts, no secrets in committed files, no personal 2FA devices handed to an agent.
  4. Required human review. Keep branch protection. Do not let an auto-approver replace a person on a repo that matters. For Copilot, plan a second reviewer and do not auto-run workflow changes.
  5. What the diff is allowed to touch. Especially CI, IaC, and dependency files.
  6. Entitlements. Seats, credits, and plan bundles differ by vendor and billing cycle. Read the current official plan page; do not assume a directory rate or a 2025 launch price still applies.

Stop the pilot if the agent can reach production secrets, if diffs merge without a human, if the only available repo is sensitive, or if official networking and retention terms are still unclear.

Who this purchase is for, and who should look elsewhere

This evaluation is for engineering teams that want a documented, isolated coding agent and are willing to review every diff.

It is not for teams that only need IDE completion or a review comment bot, and it is not a way to skip code review, security review, or repository access design.

Frequently asked questions

If the vendor says the PR is ready to merge, can we merge it? No. Official Copilot docs require thorough review and, when approvals are required, a reviewer other than the assigner. Official Codex launch guidance requires manual review of all agent-generated code. Devin’s own docs still describe pull-request workflows.

Is GitHub Copilot listed here the same as the IDE assistant? It is the same catalog tool. This article evaluates the official coding or cloud agent under that ID, not the inline-completion job.

Did you test Codex by running it? No. Codex is described from official documentation only. No Codex product, coding CLI, or agent was invoked for this review.

Why are GitLab Duo, Amazon Q Developer, and Kiro omitted? Because their official agent documentation was not opened first-hand in this review. Omission is not a ranking.

Related tools