ReadyAPI Jenkins CI Setup Guide — and a Simpler Alternative

TL;DR

Running ReadyAPI tests in Jenkins is possible through the testrunner command-line tool and the SmartBear Jenkins plugin, but it requires ReadyAPI to be installed on build agents and involves frequent configuration headaches. Apidog’s CI/CD integration works through a simple npm-installed CLI with no agent software requirements and no per-run licensing.

Introduction

Integrating API tests into a CI/CD pipeline is one of the most valuable things a testing team can do. Catching regressions on every pull request, before code reaches staging or production, is worth the setup effort.

ReadyAPI supports Jenkins integration through two mechanisms: the command-line testrunner tool and the official SmartBear Jenkins plugin. Both work, but each comes with complexity. This guide covers how to set up ReadyAPI with Jenkins, the common issues you will encounter, and how Apidog provides the same pipeline integration with less infrastructure overhead.

Setting up ReadyAPI in Jenkins: the testrunner approach

The testrunner is ReadyAPI’s command-line execution engine. When Jenkins needs to run ReadyAPI tests headlessly, it calls testrunner with arguments pointing to a project file.

Prerequisites:

ReadyAPI must be installed on every Jenkins agent that runs API tests. This is the biggest operational requirement. If you have five build agents and ReadyAPI tests run on any of them, all five need ReadyAPI installed. This creates:

• An installation maintenance burden when ReadyAPI updates.
• A licensing question: does each agent installation require its own license? SmartBear’s CI/CD licensing policy varies by contract. Confirm with your account team.
• Increased agent image size and startup time for cloud-based agents.

Basic testrunner command:

On Linux/macOS:

/path/to/ReadyAPI/testrunner.sh \
  -r \
  -f /path/to/results \
  -s "TestSuiteName" \
  -c "TestCaseName" \
  /path/to/project.xml

On Windows:

C:\ReadyAPI\testrunner.bat ^
  -r ^
  -f C:\results ^
  -s "TestSuiteName" ^
  -c "TestCaseName" ^
  C:\projects\project.xml

Key testrunner flags:

• -r: generate JUnit-compatible XML report
• -f <directory>: output directory for results
• -s <name>: run specific test suite (omit to run all)
• -c <name>: run specific test case (omit to run all)
• -e <environment>: specify environment to use
• -P <name>=<value>: override project properties

Jenkins pipeline step:

In a Jenkinsfile:

stage('API Tests') {
  steps {
    sh '''
      /opt/readyapi/testrunner.sh \
        -r \
        -f ${WORKSPACE}/test-results \
        -e ${ENVIRONMENT} \
        ${WORKSPACE}/project.xml
    '''
    junit 'test-results/*.xml'
  }
}

The junit step publishes the results to Jenkins so failures appear in the build report.

Using the SmartBear Jenkins plugin

SmartBear maintains a Jenkins plugin for ReadyAPI available from the Jenkins plugin directory. The plugin provides a build step in the Jenkins UI rather than requiring you to write shell commands manually.

To install it:

1. Go to Jenkins > Manage Jenkins > Plugin Manager.
2. Search for “SmartBear ReadyAPI Functional Testing”.
3. Install and restart Jenkins.

After installation, you configure the ReadyAPI installation path in Jenkins global settings, then add a ReadyAPI test step to your build job by selecting from the build steps dropdown.

The plugin handles argument construction for you and integrates results into Jenkins’ test reporting. For teams that prefer GUI configuration over Groovy pipeline scripts, the plugin reduces setup time.

Plugin limitations: The plugin is tied to specific Jenkins versions and ReadyAPI versions. Plugin updates lag behind both ReadyAPI releases and Jenkins releases. Teams running the latest ReadyAPI may encounter compatibility issues that require waiting for a plugin update.

Common Jenkins + ReadyAPI issues

Testrunner hangs on startup. ReadyAPI is a Java/Swing application. When launched headlessly in CI, it sometimes attempts to initialize GUI components. Set DISPLAY environment variable or use -Djava.awt.headless=true JVM arguments if you encounter this.

License validation failures. ReadyAPI validates its license at startup. If the CI agent cannot reach SmartBear’s license server (common in locked-down corporate networks), tests fail with a license error rather than a test failure. You need to configure either floating license server settings or offline licensing.

Out of memory errors. Default JVM heap settings in testrunner are conservative. Large test suites or projects with many data files may need -Xmx settings adjusted. Edit the testrunner.sh or testrunner.bat file to increase heap:

-Xms128m -Xmx1024m

Path issues with project files. ReadyAPI projects reference external files (data sources, schemas, scripts) using paths. When running in CI, relative paths work only if the working directory is set correctly. Absolute paths in project files cause issues when the project moves between machines. Use project properties for paths and set them via -P flags at runtime.

Tests pass locally but fail in CI. Often caused by environment differences: different data in external files, different environment variable values, or hardcoded paths. Use ReadyAPI’s Environment feature with explicit environment selection in the testrunner -e argument.

Apidog Jenkins integration: a simpler approach

Apidog’s CLI runner installs via npm and does not require a desktop application on the build agent.

Installation on the Jenkins agent:

npm install -g apidog-cli

Or in a pipeline step:

npm ci
# apidog-cli is in devDependencies

Running tests in a Jenkinsfile:

stage('API Tests') {
  steps {
    sh '''
      apidog run \
        --collection ${WORKSPACE}/apidog-collection.json \
        --environment staging \
        --reporter junit \
        --reporter-junit-export ${WORKSPACE}/results/report.xml
    '''
    junit 'results/report.xml'
  }
}

Key differences from ReadyAPI:

No desktop application on agents. The Apidog CLI is a lightweight Node.js package. Install it in seconds, not minutes. No GUI initialization issues, no license server connectivity requirements.

No per-run licensing concerns. Apidog does not charge per CI execution. Run as many test cycles per day as your pipeline demands.

Environment configuration via environment variables or flags. Pass credentials and base URLs as environment variables directly from Jenkins credentials store:

withCredentials([string(credentialsId: 'api-key', variable: 'API_KEY')]) {
  sh 'apidog run collection.json -e production --env-var "apiKey=${API_KEY}"'
}

GitHub Actions comparison:

For teams also using GitHub Actions, the Apidog approach is equally clean:

- name: Run API tests
  run: |
    npm install -g apidog-cli
    apidog run collection.json --environment staging
  env:
    API_BASE_URL: ${{ vars.STAGING_URL }}
    API_KEY: ${{ secrets.API_KEY }}

The ReadyAPI equivalent requires either a self-hosted runner with ReadyAPI installed or a complex Docker image build. Neither is as straightforward.

Migrating CI/CD from ReadyAPI to Apidog

If you are migrating your test suite and need to update your CI/CD configuration simultaneously:

1. Export your API definitions from ReadyAPI and import them into Apidog.
2. Install the Apidog CLI on your Jenkins agents.
3. Add an Apidog test step to your pipeline alongside the existing ReadyAPI step.
4. Run both in parallel. Compare failure rates and test coverage.
5. Remove the ReadyAPI step once you have confidence in Apidog’s coverage.
6. Remove ReadyAPI from build agents during the next agent image refresh.

FAQ

Does ReadyAPI’s testrunner require a separate CI license?SmartBear’s licensing policy for CI/CD use varies by contract. Some agreements include CI/CD execution rights in the standard license; others require a separate agreement. Check your contract or contact your SmartBear account manager before assuming CI runs are covered.

Can ReadyAPI tests run in Docker containers?Yes, with effort. You can build a Docker image with ReadyAPI installed and use it as a Jenkins agent container. The image is large (ReadyAPI is a desktop application) and requires headless configuration. It works but adds complexity compared to tools built for containerized CI environments.

Does Apidog support parallel test execution in Jenkins?Yes. You can run multiple Apidog CLI instances in parallel Jenkins stages, each running a different collection or environment. Parallel execution is controlled through your Jenkinsfile, not the Apidog CLI.

What test report format does Apidog produce for Jenkins?Apidog’s CLI can produce JUnit XML reports, which Jenkins natively parses and displays in the test results section. The format is the same as what ReadyAPI’s testrunner produces.

Do Apidog test scripts run in the same process as the CLI runner?Yes. JavaScript test scripts execute within the Apidog CLI’s Node.js process. There is no separate script execution environment to configure.

Is there an official Apidog Jenkins plugin?The Apidog CLI runs as a shell step in Jenkins, which does not require a dedicated plugin. This approach is simpler to maintain than plugin-based integrations that need updating when Jenkins or Apidog releases new versions.

ReadyAPI’s Jenkins integration works, but it requires significant infrastructure management. For teams that want to simplify their CI/CD setup while maintaining comprehensive API test coverage, Apidog’s CLI approach eliminates the most common pain points: agent software requirements, license server dependencies, and complex environment configuration.