# How to evaluate OpenAI Codex vs Claude Code

> A production-focused OpenAI Codex vs Claude Code comparison covering autonomy, safety controls, true costs, and a practical two-week trial.

OpenAI Codex and Claude Code can both produce production quality changes. The harder question is whether either one can do that repeatedly inside your repository, with permissions your team understands, evidence your reviewers can trust, and a bill you can predict. A polished demo does not answer any of those questions.

My recommendation is simple: choose the operating model before you choose the agent. Decide what the agent may read, write, execute, and reach over the network. Then make both candidates perform the same real work for two weeks. The winner is the one that closes useful tasks with fewer unsafe actions and less human repair, not the one that writes the most impressive first patch.

## The choice is an operating model, not a leaderboard

OpenAI Codex vs Claude Code is a comparison between two agent systems, not a permanent ranking of model intelligence. Models, plan limits, interfaces, and defaults change too quickly for a benchmark screenshot to settle a production decision. Your repository and controls change more slowly, so test the parts you will actually operate.

Codex spans local terminal and IDE work plus delegated cloud tasks. Claude Code centers on an interactive terminal agent and can also support scripted and remote workflows. That difference affects how work enters the queue, where it runs, and how a human resumes it. It does not make either product automatically better.

Separate four questions that teams often blur:

- Can the model reason about the task?
- Can the agent collect the right context and run the right tools?
- Can the environment contain a mistaken command?
- Can your team review and pay for the resulting workflow?

A reasoning failure produces a bad plan. An agent wrapper failure hides a needed file or loses test output. A containment failure lets an otherwise competent agent touch the wrong resource. A process failure merges code nobody truly reviewed. Calling all four "model quality" prevents you from fixing the correct layer.

Do not start with a preference poll among developers. Start with production work that already costs time: a flaky integration test, a small migration, an unfamiliar bug, a dependency update, or a review with a real regression hidden in it. An agent earns a place in the stack when it reduces the total effort to close those jobs without weakening the release gate.

The answer may be both tools. One can fit long delegated tasks while the other fits live pairing, or different repositories may need different permission boundaries. Standardize the evaluation method before you standardize the vendor.

## Autonomy means who owns the next decision

Autonomy is the number and consequence of decisions an agent can take before a person must intervene. Counting tool calls misses the point. Reading twenty files is less consequential than choosing one database migration command, and editing ten tests may be safer than sending one request with a production credential.

Codex exposes approval policy and sandbox policy as separate controls. Its current CLI reference lists approval choices such as untrusted, on request, and never, alongside sandbox scopes. It also warns that bypassing approvals and the sandbox belongs only inside an externally hardened environment. That separation is useful because human interruption and technical containment solve different problems.

Claude Code starts from a permission model in which read-only actions can run with less friction and sensitive actions request approval. Its security documentation describes working directory boundaries, sandboxed Bash with filesystem and network isolation, allowlists, and an Accept Edits mode. It also offers a flag that skips permission prompts, with an appropriately dangerous name.

The labels are not equivalent. "Auto", "accept edits", and "never ask" can grant different powers in different environments. Record capabilities, not mode names:

- Limit repository reads to declared roots.
- Limit writes to a disposable branch or isolated worktree.
- Allowlist fixed test and lint commands.
- Ask before package installation, and deny network access unless destinations are approved.
- Deny deployment tools and secrets outside a dedicated workflow with narrow task credentials.

There is no useful autonomy setting called "do whatever a senior engineer would do." A senior engineer carries years of implicit context about blast radius, ownership, and odd production history. An agent only has what the environment and instructions expose.

Set the widest safe boundary for a task class, then let the agent move freely inside it. Repeated approvals for harmless reads train people to click through. No approvals around network access and credentials turns one misunderstood instruction into an incident.

## Safety rails must contain mistakes, not decorate prompts

A prompt that says "be careful" is guidance, not a safety control. Production safety comes from denying dangerous capabilities, isolating writable state, limiting credentials, and making consequential actions pass through deterministic gates.

Use three independent layers. First, constrain the environment: a fresh worktree or ephemeral runner, no production secrets, and no broad home directory access. Second, constrain tools: fixed test commands, restricted network destinations, and no deployment client unless the task explicitly concerns deployment. Third, constrain promotion: branch protection, required tests, review ownership, and a human decision before merge.

This matters because agent failures rarely look theatrical. A common failure is a plausible command in the wrong scope. The agent runs a formatter across generated files, updates a lockfile through an unapproved registry, or "fixes" a flaky test by weakening the assertion. The patch can compile and still make the system worse.

Prompt injection deserves the same concrete treatment. Repository text, issue descriptions, logs, generated documentation, and MCP tool output can all carry instructions. Neither vendor can make untrusted text safe merely by recognizing that it looks suspicious. If the agent can read attacker controlled content and also use a credentialed network tool, you have joined data and authority in one context.

A production agent should not hold ambient cloud credentials inherited from a developer shell. Give a task only the identity it needs, expire it quickly, and log each external action. For read-only investigation, use read-only credentials. For code changes, the agent usually needs no production identity at all.

Keep the escape hatch, but make it obvious and rare. Both tools provide ways to remove prompts or containment. Those options are reasonable inside a dedicated disposable VM that supplies the real boundary. They are reckless on a laptop with SSH keys, package publishing credentials, and multiple repositories mounted.

The safety review should ask, "What can this process do if the model is wrong?" If the answer depends on the model noticing its own mistake, the rail is missing.

## Production context belongs in version control

Agents perform better when repository rules are explicit, local, and testable. A long chat prompt held by one developer is not an operating standard. Put build commands, architectural limits, generated file rules, and completion criteria where every run can read the same version.

Codex supports layered AGENTS.md instructions. Claude Code supports project memory and settings files. The filenames differ, but the useful practice is identical: keep global guidance short, put repository facts in the repository, and place narrow rules near the code they govern.

Write instructions that a reviewer can verify. "Follow best practices" has no test. "Do not edit generated clients; run make generate after changing the schema" names a boundary and a command. "Keep changes minimal" is subjective. "Do not change public API signatures in this task" is reviewable.

Good repository instructions answer these questions:

- Which command installs dependencies without updating them?
- Which tests prove the changed behavior?
- Which directories are generated or vendored?
- Which architectural boundary must the patch preserve?
- What evidence must appear in the final report?

Do not copy your entire engineering handbook into agent context. Large instruction files consume attention and accumulate contradictions. Linkless, local instructions should point to the exact commands and files needed for the current repository. Remove obsolete rules as aggressively as you remove dead code.

Treat tool configuration as code too. Review changes to allowlists, sandbox settings, hooks, MCP servers, and extra writable directories. A one line permission expansion can carry more risk than a hundred line feature patch.

Run the same context package with both agents during evaluation. If one agent receives a carefully tuned instruction file and the other receives a casual prompt, you measured prompt preparation, not the products.

## Subscription price is only the first meter

Subscription economics should be calculated per accepted change, not per seat or prompt. A cheap plan that stops during the working day, requires repeated context rebuilding, or produces expensive review is not cheap. A higher plan that remains mostly idle is not a productivity investment.

The current Claude plan structure makes the first meter easy to see: Pro at $20 per month for light use, with Max tiers at $100 and $200 for higher usage. Claude and Claude Code share plan limits, and Anthropic says actual consumption varies with repository size, conversation length, model choice, and auto-accept behavior. Teams can also move intensive work to API billing.

OpenAI now describes Codex usage through a credit rate card tied to token consumption for most current ChatGPT plans. The help material says actual credit use depends on input, cached input, output, selected model, parallel instances, automations, and fast mode. It gives a broad average of about $100 to $200 per developer each month, but that is a planning reference, not a quote for your workload.

Do not compare "$200 versus $200" and call the economics equal. The included capacity, reset behavior, shared usage, overage path, model selection, caching, and cloud execution all affect what a seat buys. Pricing can change during your evaluation, so save the vendor plan page and the date in your decision record.

Track this formula for each tool:

```text
effective cost per accepted task =
  subscription allocation
  + API or credit overage
  + reviewer minutes
  + operator interruptions
  + failed-run recovery
  + waiting cost for blocked work
```

Use fully loaded engineering cost for human time, but keep the assumptions visible. Do not turn a rough estimate into fake precision. If a reviewer costs $120 per hour and one agent saves fifteen review minutes per task, record the arithmetic. If you cannot measure the time, mark it unknown.

Subscription caps create a scheduling problem as well as a cost problem. An agent that exhausts shared capacity during a release can push work to a more expensive API path or leave engineers waiting. Test the heavy day, not only the average day. Run parallel sessions if that reflects normal use and watch whether one person's work consumes capacity needed elsewhere.

The economically stronger tool is the one that lowers total delivery cost at an acceptable risk level. Token price matters, but repair time and queue delay can dominate it.

## Evaluate completed work instead of prompt demos

A production evaluation begins with a frozen task brief and ends with independently checked evidence. Do not reward fluent explanations, attractive terminal output, or a patch that merely looks large. Reward correct behavior, bounded changes, and a reviewable account of what happened.

Use tasks with known acceptance criteria but do not reveal the reference solution. Each task should have a clean starting commit, a time limit, permitted tools, and the same repository instructions. Give both agents the same initial prompt. When an agent asks a legitimate clarifying question, answer both runs with the same information.

Keep humans from rescuing one candidate more than the other. Record every intervention: clarification, permission approval, manual command, prompt correction, and code edit. A five prompt success is different from a one prompt success, even if the final diffs match.

After each run, capture a small evidence bundle. These commands work in any Git repository and do not depend on either agent:

```sh
git status -s
git diff
```

A clean status has no output. Otherwise it prints a two character state followed by each changed path, such as " M src/parser.ts". The diff shows every unstaged patch with file headers and changed lines. Save that output, the staged diff if one exists, test output, agent transcript, elapsed time, and usage reading under a run identifier.

Do not let the agent grade itself. A human or deterministic test suite must decide whether behavior is correct. Agent summaries are useful indexes, but I have seen confident summaries omit a changed file and claim tests passed when only a subset ran.

Blind review helps when team preference is strong. Remove vendor names from the evidence bundle, then ask reviewers to score correctness and maintainability before revealing the agent. You may discover that the favored interface did not produce the favored code.

## A fixed task set exposes different failure modes

The task set should cover the work you expect to delegate and the failures you cannot afford. Five carefully chosen tasks tell you more than twenty toy exercises because each one can target a distinct production behavior.

Use one task from each relevant class:

1. Diagnose an unfamiliar defect with a failing regression test. This measures repository search, causal reasoning, and restraint.
2. Make a small cross-file feature under an existing architecture. This tests planning, consistency, and whether the agent invents abstractions.
3. Upgrade a dependency with a lockfile and a breaking change. This exposes network assumptions, documentation use, and generated churn.
4. Review a prepared patch containing a subtle security or data integrity defect. This tests whether the agent finds consequences beyond style.
5. Repair a CI-only failure from logs without access to the original environment. This measures uncertainty handling and evidence discipline.

At least one task should tempt the agent into an unsafe shortcut. Put a production-looking deployment script in the repository but forbid its execution. Include a flaky test whose assertion must not be weakened. Place generated code beside the source schema. The goal is not to trick the model with word games. The goal is to reproduce the choices that cause real review pain.

Use medium-sized tasks that a competent engineer can finish and review within the trial. Huge migrations create too many confounding factors. Trivial functions measure code completion, not agent work.

Define acceptance before the run. For a bug, require the new regression test to fail on the base commit and pass on the patch. For an upgrade, require the intended version, unchanged unrelated dependencies, and the full relevant suite. For review, maintain a hidden list of seeded defects and penalize invented findings.

Rotate repositories if your team owns different stacks, but keep at least one repository constant across all participants. Otherwise language familiarity can masquerade as vendor advantage. If one agent cannot access a required environment under the approved safety policy, count that as an operational result rather than silently relaxing the boundary.

## A two-week trial should resist moving goalposts

Two weeks is enough to expose workflow friction if you schedule the trial before seeing results. It is not enough to prove every production case, so the output is a bounded adoption decision with explicit unknowns.

Use this sequence.

### Days 1 to 5

1. Freeze tasks, base commits, acceptance tests, permission rules, and scoring weights. Save a signed evaluation brief.
2. Install both agents in equivalent isolated environments. Record every version and configuration value.
3. Run one calibration task that does not count toward the winner. Correct symmetric setup errors and save the notes.
4. Run defect diagnosis with no manual edits. Save the patch, transcript, tests, usage, and interventions.
5. Run the feature task, then review both results blind. Record reviewer scores before revealing the agent.

### Days 6 to 10

1. Run dependency work under the declared network policy. Capture approvals, the diff, and test evidence.
2. Run seeded code review and CI diagnosis. Compare true findings, false findings, and recovery paths.
3. Repeat the weakest task class with fresh cases. Look for consistency or a repeated failure pattern.
4. Run expected parallel load. Record capacity, queue time, resets, and overage behavior.
5. Reconcile costs, risks, and reviewer feedback. Write the adoption decision and its rollout boundary.

The calibration day is important. Installation trouble, missing build tools, or a bad shared instruction can spoil the comparison. Fix symmetric setup errors before scored work begins. Do not tune away a product-specific limitation unless the same type of adjustment is available to both.

Assign one evaluation owner. Developers can run tasks, reviewers can score patches, and security can inspect boundaries, but one person must keep base commits, prompts, answers, and artifacts consistent. Without that role, the loudest anecdote wins.

Lock scoring weights on day one. A regulated team may give safety violations more weight than speed. A small product team may value task completion and reviewer time, while still treating secret access as an automatic failure. Both are defensible. Changing weights after a favored tool loses is not.

Record versions because both products update quickly. Do not upgrade one agent halfway through unless a blocking defect forces it. If you must upgrade, rerun affected tasks on both current versions or mark the comparison compromised.

Day ten should end with one of four decisions: adopt for defined task classes, extend the trial to answer named unknowns, keep both for distinct workflows, or adopt neither. "The team liked it" is not a decision record.

## Score evidence and make hard failures explicit

Averages can hide unacceptable behavior. A tool that finishes four tasks quickly and exposes a secret on the fifth should not beat a slower tool through arithmetic. Define hard failures before assigning points.

Reasonable hard failures include writing outside the approved workspace, attempting a prohibited deployment, reading a secret path, using an unapproved network destination, changing tests to conceal broken behavior, or claiming a required test passed when it did not run. Decide whether an attempted action blocked by the sandbox counts differently from a completed action. I score the attempt as a reasoning failure and the block as a safety success.

For runs without a hard failure, score these dimensions on a five point scale:

- Correctness against hidden acceptance tests
- Scope discipline in changed files and dependencies
- Evidence quality in commands, output, and final report
- Human effort across prompts, approvals, review, and repair
- Cost and capacity under the actual account meter

Use behavioral anchors. A correctness score of five means all acceptance checks pass and reviewers find no functional defect. A three means the main behavior works after a small human correction. A one means the approach is wrong or the patch cannot be safely repaired within the task limit.

Store raw facts beside scores. This compact record is enough to reconstruct a decision:

```yaml
run_id: defect-a-candidate-1
base_commit: 2f4c1ab
agent_version: recorded-at-run
task_result: accepted
hard_failures: []
elapsed_minutes: 38
human_interventions: 2
files_changed: 4
tests:
  required: 18
  passed: 18
usage:
  meter: vendor-reported
  amount: recorded-at-run
scores:
  correctness: 5
  scope_discipline: 4
  evidence_quality: 5
  human_effort: 3
  cost_capacity: 4
```

Do not combine numbers until reviewers resolve factual disagreements. If one reviewer says the migration is safe and another identifies possible data loss, investigate the migration. Averaging their opinions to four out of five launders uncertainty.

Read transcripts for patterns after scoring. Frequent premature edits, ignored instructions, repeated broad searches, and self-corrections explain why a run consumed time. They also tell you whether better repository guidance could fix the issue or whether the agent conflicts with your workflow.

## The right fit depends on how your team ships

Choose Codex when its mix of local work, cloud delegation, permission profiles, and your existing OpenAI account economics fits the task queue you want to build. Choose Claude Code when its terminal interaction, permission system, hooks and tool controls, and Claude subscription or API path produce better accepted work under your constraints.

That is intentionally conditional. Claims that one agent always plans better or the other always writes better code decay with the next model release and usually ignore repository language, task type, and prompt quality. Your own blinded results have more value than a public benchmark built on unrelated tasks.

Interface fit still matters. A developer who works by steering an agent continuously may prefer a responsive terminal loop. A team that wants to assign work and inspect results later may value delegated execution. Measure both the active collaboration time and the unattended completion rate. Do not confuse less visible work with less total work.

Team design matters more than individual preference once several agents run in parallel. You need ownership of branches, limits on concurrent changes to the same files, a queue for review, and a way to stop duplicate work. Two agents editing the same migration can waste more time than either saves.

A mixed setup can be sensible, but it carries costs: two policy systems, two instruction formats, separate usage meters, and duplicated training. Keep both only if each owns a clear task class or provides meaningful fallback capacity. "Some developers like each one" is too weak a reason.

In my Team & AI Audit work, I look for the workflow that removes expensive waiting and rework, then set the agent boundary around it. Tool choice follows the operating constraint; it does not substitute for one.

## Make adoption reversible and review it with data

Adopt the winner for the task classes it proved, under the exact safety boundary used in the trial. Do not turn a successful bug-fix evaluation into immediate permission to run migrations, publish packages, or touch infrastructure.

Start with a written production policy that names allowed repositories, writable roots, network rules, credential rules, required tests, reviewer ownership, and usage budget. Pin or record agent versions. Keep transcripts and evidence according to your code retention policy. Train reviewers to verify commands and changed files rather than accepting the final summary.

Set a review date and triggers for an earlier review. A major model change, pricing change, new sandbox behavior, repeated capacity exhaustion, or a safety incident can invalidate the decision. The comparison is a maintained engineering decision, not a procurement ceremony.

Watch four operating signals after rollout: accepted tasks per week, median human intervention time, rollback or repair work, and cost per accepted task. Add safety events separately because an average must not hide them. Compare against the baseline work process, not against a fantasy of fully autonomous engineering.

If neither candidate clears the hard-failure bar, keep the useful pieces. Teams can still use read-only planning, code explanation, or draft review while denying edits and commands. Restricted value is better than granting broad authority to force an adoption story.

The first production action after the trial should be boring: enable one proven task class for a small group, with the same branch protection and evidence capture that produced the result. If the workflow remains safe and economical under normal load, widen it deliberately.