How to Build Your Own Software Factory With Codex, CI, and Apidog API Tests

Gergely Orosz’s diagram of OpenAI’s internal software factory has been making the rounds this week: a builder files an outcome, Codex writes the code, a fleet of specialist review agents argues about risk, and an agent babysits the deploy while watching dashboards it built itself. It is also, for the parts that matter most to OpenAI’s own engineers, a description of internal tooling you cannot install.

Perf Factory, Sevbot, and agentic deploy with self-built dashboards run on OpenAI’s own observability stack and are not part of the Codex product you can buy. What you can build this week is a smaller loop that still does real work: a builder defines the outcome as an issue, Codex works the branch, CI checks that the change actually behaves, one or two reviewer agents look at it, and a human signs off on anything risky before it ships behind a flag. The whole thing turns on one detail the original diagram glosses over: “CI passes” only means something if CI checks the right things. For an API, that means testing the API, not just the code that calls it.

What the diagram gets right (and what you cannot copy)

The public part of the Pragmatic Engineer piece describes a core loop where Codex “makes a series of code changes until it reaches its goal, and then verifies that the software works as it should.” Low-risk areas can auto-approve; higher-risk changes get more AI review or a mandatory human pass. That structure, propose, verify, review by risk tier, is portable. Teams have approximated it with pull request bots and CI gates for years; agents just make the loop faster and less supervised.

What is not portable is the internal machinery around it. Perf Factory sifts alerts and dashboards to find latency regressions and propose fixes. Sevbot investigates incidents and answers questions in Slack, though it does not execute mitigations itself. Agentic deploy watches a change into production and builds its own monitoring using OpenAI’s internal telemetry stack. None of those three ship in the external Codex product. What did ship is the desktop app, the /goal command for long-running tasks, and role plugins and skills you configure yourself. OpenAI’s Codex product page covers what is actually available.

The five-step loop you can run this week

A scaled-down version looks like this:

1. A builder files the outcome as an issue. Not a task list, a description of the end state: “orders can carry an optional discount code that reduces the total.” Handing that same GitHub issue to an Agent in Sharkly is the shipped integration: the issue becomes a Task, and the run comes back as a PR rather than a separate ticket to reconcile. It’s currently free for organizations up to 10 people.
2. Codex works the branch with /goal. As covered in how the /goal command drives autonomous Codex and Claude Code runs, you give the agent a target and let it iterate on its own until the target is met.
3. CI runs build, unit tests, and API test scenarios. This is the step most teams skip or half-build.
4. One or two reviewer agents check the diff, and a human reviews anything above low risk. AI code review tools can catch a lot before a human ever opens the PR. In Sharkly, this is where Ready for Release does the work: a human has to move the task out of that status before anything reaches Done, and a separate reviewer Agent can sit in the same Crew as the one that wrote the code.
5. Deploy behind a feature flag, so a bad merge is a toggle, not an incident.

Step 3 is where the loop either works or lies to you.

Why CI has to test the API, not just the code

Codex’s own review feature, covered in how Codex code review works, reads the diff and flags obvious problems. It does not run your service and check what it returns. Unit tests, if the agent wrote or kept them, mostly verify the code does what the code intends, which is not the same as verifying the API does what the contract promises. An agent editing a handler can pass every unit test while quietly breaking the response every client depends on.

Say you run an orders API. POST /api/orders creates an order and returns its record; GET /api/orders/{id} fetches one by ID. You maintain an OpenAPI spec for both, and you built Apidog test scenarios against it: create an order, fetch it back, and check four things a unit test typically will not:

• Status codes. POST /api/orders returns 201, not 200 or a silent 500 on a validation edge case.
• Response schema against the OpenAPI spec. The total_amount field stays a number, status stays one of the enum values you defined, and no field that clients depend on quietly disappears or gets renamed.
• Auth failures. A request without a valid token returns 401, not a 200 with an empty body, which is a surprisingly common regression.
• A latency threshold. The scenario asserts the response comes back under a set limit, so a change that adds an unbatched database call inside the handler gets caught before a customer notices.

Those are exactly the checks a handler-level refactor can silently break while every unit test still passes, because unit tests usually mock the boundary the API scenario actually exercises.

Wiring it into the pipeline

You already run the CLI version of these scenarios locally if you followed how to use the Apidog CLI in Codex. The same command runs in CI. A GitHub Actions job that builds, runs unit tests, and then runs the Apidog scenario looks like this:

name: CI

on:
  pull_request:
  push:
    branches: [main]

jobs:
  build-test-verify:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Build and run unit tests
        run: npm run build && npm test

      - name: Install Apidog CLI
        run: npm install -g apidog-cli

      - name: Run orders API test scenario
        env:
          APIDOG_ACCESS_TOKEN: ${{ secrets.APIDOG_ACCESS_TOKEN }}
        run: |
          apidog run \
            --access-token $APIDOG_ACCESS_TOKEN \
            -t 88214 \
            -e 3301 \
            -r cli,junit

      - name: Upload Apidog reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: apidog-reports
          path: apidog-reports/

The -t and -e values are your real scenario and environment IDs from Apidog, not placeholders you invent. The apidog run command reference covers every flag, and Apidog CLI test reports explains the JUnit output the job uploads. apidog run exits non-zero on any failed assertion, so GitHub Actions marks the job failed the same way it would for a failed unit test.

What the agent’s prompt looks like

The point of /goal is that you describe the outcome and the exit condition, and Codex iterates without you approving every step. For the discount code example, a reasonable prompt is:

/goal Add an optional `discount_code` field to POST /api/orders. Validate it
against the promotions service and apply the discount to `total_amount` in
the response. Do not rename or remove any existing response field. Run
`npm test` and `apidog run --access-token $APIDOG_ACCESS_TOKEN -t 88214 -e 3301
-r cli` before you finish. Both must exit 0. If the Apidog run fails, read the
failing assertion and fix the handler, not the test.

That last line matters. An agent under pressure to turn a check green will sometimes edit the assertion instead of the bug. Telling it explicitly which side of the loop to fix keeps the test scenario as the source of truth, not an obstacle to route around.

When the agent breaks the contract

Codex adds the discount field, and in the process renames total_amount to totalAmount because that is the convention in a file it read nearby. Unit tests still pass; they check the discount math, not the field name. The build succeeds. Then the Apidog scenario runs in CI, validates the response against the OpenAPI spec, and fails: the spec says total_amount, the response now has totalAmount, and the schema assertion catches it immediately.

CI reports a non-zero exit and points at the failing schema assertion in the JUnit output. Codex reads the failure, sees the rename is the cause, and reverts it while keeping the discount logic. The scenario passes, the build goes green, and the pull request moves to review with an actual guarantee behind the word “passing.” Without the API-level check, that rename ships, and every client parsing total_amount breaks on the next release.

Risk tiering you can ground in the spec

Instead of a vague sense of what counts as low risk, tie your risk classification to the OpenAPI diff. A change that adds an optional field with a default is a candidate for auto-merge once tests pass. A change that removes a field, renames one, or changes a status code is never low risk, regardless of what the rest of the diff looks like. That single rule catches most of what a specialist review agent would flag anyway. Route anything the rule flags to a human reviewer or a second pass from an AI code review tool before merge.

Ship behind a flag, not into the void

Once a change clears CI and review, deploy it behind a feature flag rather than straight to every user. This is the cheap substitute for OpenAI’s agentic deploy step: no agent babysitting the rollout or building its own dashboards. A flag that starts at 5% traffic and a person who checks error rates before flipping it to 100% gets you most of the safety with none of the internal tooling. If something is wrong, you flip the flag off instead of rolling back a merge under pressure.

A scaffold that already exists

You do not have to wire all five steps from scratch. orchflows is an open-source project that showed up in the replies to Orosz’s thread: an MIT-licensed /software-factory command for Claude Code and Codex, built around a small number of reusable skills. It is a starting scaffold, not a replacement for the CI and review steps above; you still point it at your own test scenarios and risk rules.

What to skip building

Do not try to reproduce Perf Factory, Sevbot, or agentic deploy with self-built dashboards. They are internal OpenAI systems plugged into telemetry most teams do not run. Humans at OpenAI still define outcomes, approve high-risk changes, authorize incident mitigations, and hold oncall; as OpenAI put it, “oncall duty is not a thing of the past.” Copy the parts of the loop that are just good engineering discipline: verify before merging, tier risk by what actually changed, and keep a human on anything not obviously safe.

Get the loop running

Start with the CI step; it makes every other step trustworthy. Build your orders API test scenario in Apidog, covering status codes, schema, auth, and a latency budget. Wire it into your pipeline with the CLI, point /goal at a real issue, and let Codex iterate against a check that actually asserts API behavior instead of trusting the agent’s word for it. Download Apidog to build the first scenario, then add the reviewer and flag steps once the loop proves itself.