The Golden Rule: CLI Produces Facts, Model Acts on Facts

This is a 10-part series sharing how Apidog developed Apidog CLI, a command-line tool for API testing and API lifecycle management. Read in order or jump to any post that interests you:

Don't make the model memorize all rules—let rules be executed at the right places. cli-schema validate turns Schema from knowledge into a quality gate.

The Core Principle: Let Rules be Executed at the Right Places.

We distilled one core principle from our experience:

Don't make the model memorize all rules. Let rules be executed at the right places.

This is similar to a lesson from Agent evaluation:

In Apidog CLI + SKILL:

Let CLI validate structure. Let Agents generate content.

The Problem with Model Memory

When an AI Agent helps create or update Apidog resources, the risky part isn't just generating content.

The risky part is writing generated content into a real project without enough structure or verification.

Apidog resources are structured. Consider what a test case or test scenario includes:

If an Agent guesses the structure:

• Wrong field name → Failed write
• Invalid enum value → Server rejection
• Missing required field → Incomplete resource
• Wrong type → UI display issues
• Incorrect nesting → Tests don't behave as expected

cli-schema validate: The Quality Gate

The most direct embodiment of our principle is cli-schema validate.

apidog cli-schema validate test-scenario-update --file ./scenario-update.json

When an Agent wants to write or update a test scenario, having AI generate complex step structures is very error-prone.

The validate command:

• Confirms field names
• Checks structural validity
• Verifies enum value effectiveness
• Validates type constraints

All before initiating the write request.

Common Errors cli-schema Catches

Here are real examples of errors Agents commonly make—and that cli-schema validate catches:

These aren't theoretical. We discovered them through real Agent interactions.

Each error would cause:

• A failed write request
• An API error response
• Agent confusion about what went wrong
• Multiple retry attempts
• Token waste on repeated calls

With cli-schema validate, these errors are caught locally, before the network call.

The Design Philosophy

Let's consider the alternatives:

Alternative 1: Write Rules into Prompt

If we wrote all field rules into the Agent's prompt:

• Every field name documented
• Every enum value listed
• Every type constraint explained
• Every nested structure described

Result: Massive context burden.

A comprehensive test scenario schema could easily require 5,000+ tokens of description. That's context the model must carry for every task, even when most rules aren't relevant.

Alternative 2: Rely on Model Memory

If we rely on the model to "know" the correct structure:

• Model trained on some API patterns
• But not specifically on Apidog schemas
• Field names vary across products
• Enum values are product-specific

Result: High error rates.

The model doesn't have perfect memory of Apidog-specific conventions. It will guess—and guesses will be wrong.

Better Approach: Validate Locally

Let the Agent generate drafts. Let CLI execute validation before writing.

# Agent generates JSON
# (Agent doesn't need to memorize all rules)

# CLI validates
apidog cli-schema validate test-case-create --file ./test-case-create.json

# CLI outputs specific errors if any
# Agent adjusts based on errors

# Only valid writes proceed
apidog test-case create --project <projectId> --file ./test-case-create.json

Schema Transformation

cli-schema validate transforms what Schema means:

Problems are not consumed in meaningless back-and-forth network requests.

Quality checks are completed through local commands.

Practical Example

Let's walk through a real workflow:

# Agent reads endpoint
apidog endpoint get <endpointId> --project <projectId>

# Agent generates test case JSON
# (Creates ./test-case-create.json)

# Validate before writing
apidog cli-schema validate test-case-create --file ./test-case-create.json

If validation passes:

apidog test-case create --project <projectId> --file ./test-case-create.json

If validation fails:

Error: Field "assertions[0].comparator" has invalid value "contains"
Valid values: equal, not_equal, greater, less, include, not_include, exists, not_exists

Error: Field "extractors[0].type" has invalid value "global"
Valid values: globals, environment, collection, local

Suggestion: Fix these fields and re-validate before writing.

The Agent:

1. Reads the specific errors
2. Understands exactly what's wrong
3. Adjusts the JSON file
4. Re-runs validation
5. Proceeds only when valid

No failed writes. No confused retries. No wasted tokens.

The Broader Lesson

This principle extends beyond validation.

Deterministic rules → Engineering system

Semantic judgment → Agent

What's Next

Now that we've established the validation principle, the next question is:

After validation, how does CLI guide the Agent to the next step?

In Part 4, agentHints: Teaching CLIs to Talk to Agents, we'll explore how structured output with next-step suggestions transforms CLI from a command executor into a workflow navigator.

Key Takeaways

• Core principle: rules belong in execution, not in context
• cli-schema validate is the quality gate before writing
• Common errors: wrong field names, invalid enums, wrong types
• Validation catches errors locally, saving network round-trips
• Schema transforms from "knowledge to memorize" to "gate to pass"
• Deterministic rules → engineering; semantic judgment → Agent

Download Apidog to design, mock, test, and document APIs in one workspace. Learn more about Apidog CLI for command-line API testing, CI automation, and AI Agent workflows.