Apidog CLI vs Newman: Run API Tests in CI (2026)

Your API tests pass on your laptop. The harder question is whether they run on every pull request, every merge, and every nightly build without a human clicking anything. That job belongs to a command-line runner. It takes the tests you already wrote, executes them headlessly inside your pipeline, exits with a status code your CI can read, and writes a report your dashboard can parse. No GUI, no manual step, no excuse to skip the run.

For years the default answer to that need has been Newman, Postman’s open-source command-line collection runner. It is a solid, well-understood tool, and plenty of teams still reach for it first. But if you author your tests in Apidog instead of Postman, you have a second option: the Apidog CLI, which runs the same test scenarios you build visually in the app, headlessly in CI. This article is an honest, command-level comparison of the two. It covers what Newman does well, where the Apidog CLI fits, and how each one feels once it is wired into a real pipeline.

TL;DR

• Newman (newman, installed via npm install -g newman) runs an exported Postman collection JSON file. It is open source, free, and reads a collection plus an environment file from disk or a URL. It supports environments, data files, iteration counts, and JUnit/JSON/HTML reporters, and it exits non-zero when a test fails.
• Apidog CLI (apidog-cli, binary apidog) runs the test scenarios you design in the Apidog app, fetched by ID using an access token. You do not export anything or maintain a JSON file by hand; the visual scenario is the test, and the CLI runs it exactly as the app would.
• Both plug into GitHub Actions, GitLab CI, Jenkins, and anything with Node.js. Both fail the build on a failed test. JUnit XML is what plugs into CI dashboards on either side.
• Pick Newman when your tests already live as Postman collections and you want a free, repo-friendly runner with no new account.
• Pick Apidog CLI when you want visual authoring, request chaining, environment management, and data-driven runs that stay in sync with the same scenario your team debugs every day.

The real problem: tests that exist but never run

A test you run by hand is a test that rots. Someone built it, it passed once, and then it sat untouched while the API drifted underneath it. More tests do not fix that. Tests that run automatically on every change do, because they produce a pass or fail signal your pipeline can act on.

A CLI runner closes that gap. To be useful in CI, it has to do three things: run without a GUI, exit non-zero when something fails so the build goes red, and write a machine-readable report so reviewers can see what broke. Newman and the Apidog CLI both clear that bar cleanly. Where they differ is upstream of the run command, in how the test got written and where it lives. If you are setting up CI from scratch, the broader patterns in how to automate API tests in CI/CD pair well with this comparison. Here we stay focused on the two runners.

What Newman does well

Newman has earned its place. It is the official open-source companion to Postman, and for teams whose tests already exist as Postman collections, it is the most direct path from “tests on my machine” to “tests in CI.” It is worth stating its strengths plainly before any comparison.

It is free and open source. You install it from npm and run it anywhere Node.js runs, with no separate license for the runner itself. Your collection is a portable JSON file you can commit to a repo, store as a build artifact, or fetch from a URL. That portability is real, and it means Newman drops into almost any pipeline without friction.

It reuses what you already built. If your team writes requests, pre-request scripts, and test assertions in Postman, Newman runs those same collections unchanged. There is no rewrite. The scripts you wrote in the Postman editor execute the same way in the headless run, which keeps the learning curve flat for Postman shops.

Installing it is one command:

npm install -g newman

The binary is newman. You run an exported collection by pointing it at the JSON file, usually with an environment file alongside it:

newman run api-tests.postman_collection.json -e staging.postman_environment.json

That is the core loop. Export the collection from Postman, commit the JSON, and run it. Newman reads the requests and assertions from the file and executes them in order. For the full setup path, Apidog’s own walkthrough on how to install Newman and run Postman collections covers the export-and-run flow step by step.

Newman also supports the CI essentials you would expect. Data-driven runs come from a CSV or JSON file:

newman run api-tests.postman_collection.json \
  -e staging.postman_environment.json \
  -d users.csv \
  -n 5

Here -d supplies the data file and -n sets the iteration count, so the collection runs once per row or a fixed number of times. These are the same primitives any serious runner needs, and Newman has had them for years.

Where Newman starts to cost you

The strengths above are honest, but so are the trade-offs, and they show up in day-to-day maintenance rather than in any single command.

The biggest one is the export step. A Postman collection in CI is a snapshot. You author and debug in the Postman app, then export a JSON file to run headlessly. The moment someone edits a request in the app and forgets to re-export, your committed collection drifts from the source of truth. The CI run keeps passing against an old definition while the real API moved on. Nothing in the tooling forces the two back into sync; it is on whoever remembers to export.

Scripting is the other one. Postman test logic lives in JavaScript snippets attached to each request, and those scripts are where chaining, variable extraction, and assertions happen. That is flexible, but it means your test suite is a pile of small scripts to write, debug, and maintain by hand. As scenarios grow, so does the script surface you own. This is part of a wider pattern teams hit as collections scale, which we go into in Postman Collection Runner restrictions and how to work around them.

None of this makes Newman a bad tool. It makes it a runner whose ergonomics are tied to Postman’s authoring model: scripts plus an export step. If that model fits your team, Newman is fine. If the export-and-sync dance is the part that keeps breaking, that is exactly where a different runner helps.

What the Apidog CLI does differently

Apidog takes a different path to the same pipeline. You build tests visually in the Apidog app. You chain requests into a test scenario, add assertions in a no-code editor, pull a value from one response into the next request, and loop the whole thing over a data file. The CLI is the headless executor for those scenarios. It has no separate file format and nothing to export. It fetches the scenario you name by ID from your Apidog project and runs it exactly as the app would.

That removes the drift problem. The scenario in the project is the test. There is no exported snapshot to fall out of sync, because the CLI runs the live scenario, not a copy of it. You author it once in a visual builder that handles request chaining and assertions for you, then the same scenario runs in CI. The fast authoring loop and the automation loop share one source of truth.

Install is one npm command:

npm install -g apidog-cli

The binary is apidog. A typical run names a scenario by ID, picks an environment, and authenticates with an access token:

apidog run --access-token $APIDOG_ACCESS_TOKEN -t 605067 -e 1629989 -n 1 -r html,junit

You do not type those IDs by hand. Open the test scenario in Apidog, switch to its CI/CD tab, generate an access token, and Apidog builds the full command for you with the scenario ID and environment ID already filled in. You copy it once, then move the token into a CI secret and reference it as $APIDOG_ACCESS_TOKEN. If you are not sure which flags your installed version supports, run apidog run --help and it prints the exact options available.

The token-based, fetch-by-ID model is the clearest difference from Newman. Newman reads a collection file from disk; the Apidog CLI reaches into a project over the network, authenticated, and runs the scenario stored there. Neither is wrong. They suit different team setups, and the choice is mostly about whether you want the test to live as an exported file or as a live scenario in a shared workspace.

Side by side

Two things stand out. First, both runners cover the same CI essentials: environment selection, data-driven iteration, the report formats that matter, and a non-zero exit on failure. The flag names are even close enough that muscle memory transfers (-e, -d, -n, -r). Second, the real split is not raw capability. It is where the test lives and how you wrote it. Newman runs an exported collection authored with scripts. Apidog runs a live visual scenario by reference. The deeper version of this comparison, framed around the two Postman-world runners, lives in Postman CLI vs Newman if you want that angle too.

Reporters and exit codes: the parts CI actually reads

A runner earns its place in a pipeline through two behaviors: the report it writes and the exit code it returns. Get these right and the rest is wiring.

Newman selects reporters with the -r flag and a comma-separated list. JUnit and JSON ship in the box; the HTML reporter is the most common add-on:

newman run api-tests.postman_collection.json \
  -e staging.postman_environment.json \
  -r cli,junit \
  --reporter-junit-export ./results/junit.xml

The JUnit XML is what your CI dashboard parses into a pass or fail tree. Newman’s --bail flag stops the run after the first failure, which keeps feedback fast on a smoke test. Without it, the run completes and reports every failure at once.

The Apidog CLI uses the same comma-separated -r flag and writes everything under one output directory:

apidog run --access-token $APIDOG_ACCESS_TOKEN -t 605067 \
  -r html,junit --out-dir ./apidog-reports

Its --on-error flag shapes mid-scenario behavior: end stops at the first failure and is the default, continue runs every step so you collect all failures in one report, and ignore skips past a known-flaky step. Either way the run ends non-zero if anything failed.

The exit-code contract is the same on both sides, and it is the load-bearing part. When an assertion fails, the runner exits non-zero. CI reads that code, marks the step failed, fails the job, and blocks the merge or deploy. You configure nothing extra. The one trap, identical for both, is swallowing the exit code: wrap the run in a shell pipeline or append || true and the non-zero exit gets eaten, so the gate silently stops working. Do not do that. For the wider context on report formats, data-driven API testing with CSV or JSON shows how the report ties back to the iteration data.

Newman in GitHub Actions

Because the collection is an exported file in the repo, the workflow is short. Check out the code, install Newman, run the collection, upload the report even on failure.

name: API tests

on:
  pull_request:
    branches: [main]

jobs:
  api-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Newman
        run: npm install -g newman

      - name: Run API tests
        run: |
          newman run ./tests/api-tests.postman_collection.json \
            -e ./tests/staging.postman_environment.json \
            -r cli,junit \
            --reporter-junit-export ./results/junit.xml

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: newman-report
          path: ./results

No secret is needed if the collection and environment files are committed to the repo. The if: always() line keeps the report upload running even when tests fail, which is exactly when you want to read it. The cost you are signing up for is the export step that produced those JSON files in the first place, and remembering to refresh them when the API changes.

Apidog CLI in GitHub Actions

The Apidog workflow is the same shape, with one change: the access token comes from repository secrets, and you select the scenario by ID instead of pointing at a file.

name: API tests

on:
  pull_request:
    branches: [main]

jobs:
  api-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Apidog CLI
        run: npm install -g apidog-cli

      - name: Run API test scenario
        run: |
          apidog run \
            --access-token "$APIDOG_ACCESS_TOKEN" \
            -t 605067 \
            -e 1629989 \
            -r html,junit \
            --out-dir ./apidog-reports
        env:
          APIDOG_ACCESS_TOKEN: ${{ secrets.APIDOG_ACCESS_TOKEN }}

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: apidog-report
          path: ./apidog-reports

Notice the symmetry. Same checkout, same Node setup, same install-then-run shape, same always-upload. The only real differences are the token wired in as a secret and the scenario selected by ID rather than by file path. Because the scenario is fetched live, there is no JSON file to keep current. For the same pattern in other systems, how to automate API tests in GitHub Actions carries the structure across GitLab CI and Jenkins.

How to choose

The decision rarely comes down to which runner is objectively better. It comes down to where your tests already live and how your team wants to maintain them.

Choose Newman when your tests are already Postman collections. If your suite lives in Postman, your team writes test scripts in the editor, and you want a free, open-source runner that drops into any pipeline with no new account, Newman is the direct fit. It is the natural pick for Postman shops that are comfortable exporting a collection and committing the JSON, and that do not mind owning the test scripts by hand. You can read more on the differences inside the Postman ecosystem in what is the difference between Newman and Postman.

Choose the Apidog CLI when authoring speed and a single source of truth matter more than staying inside Postman. If you would rather build a test scenario visually, chain requests, extract variables, and run that same scenario across environments without writing and re-exporting test code, Apidog removes that friction. Design, debug, mock, and test live in one workspace, and the CLI runs the exact scenario you built. For teams coming off Postman, the migration is one Apidog covers as a Postman alternative for API testing, and the import is one click.

There is also a both-tools answer. Some teams keep an existing Newman step running their legacy collections while they move new, more complex chained scenarios into Apidog. The two CLIs coexist fine in one pipeline; they are separate steps with separate exit codes, and you can retire the Newman step on your own timeline.

To set up your first automated scenario and run it from the terminal the same afternoon, download Apidog, build a test scenario, and copy the generated command from its CI/CD tab.