How to Find the API of a Website with AI

Want to uncover the hidden APIs powering your favorite websites? With Hyperbrowser, an AI-powered platform for web automation, you can find API endpoints in seconds—no more digging through code like a detective. I used Hyperbrowser to spot APIs on a site like retouched.ai in ~60 seconds! In this tutorial, we’ll explore how to use Hyperbrowser’s deep-crawler-bot to discover website APIs, plus a classic non-AI method using browser Developer Tools. Let’s dive into the Hyperbrowser magic!

What is Hyperbrowser for API Discovery?

Hyperbrowser is a platform for running headless browsers in secure, isolated containers, built for AI-driven web automation and API discovery. Its DeepCrawler tool, powered by AI, scans websites to identify API endpoints, bypassing anti-bot measures like CAPTCHAs. Key features include:

• AI-Driven Scanning: Automatically detects API calls (e.g., XHR/fetch requests).
• Stealth Mode: Evades bot detection with proxies and CAPTCHA-solving.
• Structured Output: Exports endpoints to JSON, Postman, or OpenAPI specs.
• Integration: Works with LangChain, SDKs, and AI agents for automation.

Hyperbrowser’s DeepCrawler is praised for finding APIs in under a minute, with a clean UI and mobile support. Ready to hunt APIs? Let’s go!

Why Use Hyperbrowser for API Discovery?

Hyperbrowser makes API discovery a breeze compared to manual methods:

• Speed: AI scans sites in seconds, versus hours of manual inspection.
• Accuracy: Identifies hidden APIs (frontend- or backend-tied) reliably.
• Ease: No coding expertise needed—just a simple setup and prompt.
• Privacy: Runs locally or in secure containers, keeping your scans private.

Compared to Google Analytics’ tracking APIs, Hyperbrowser focuses on actionable endpoint discovery for developers. I found an API on a photo-editing site faster than I could brew coffee!

How to Find the API of a Website: Step-by-Step Guide

Let’s find the API of a website like retouched.ai (e.g., for background removal) using Hyperbrowser’s deep-crawler-bot and the traditional Developer Tools method. You’ll need Node.js, a Hyperbrowser API key, and a browser. No API guru status required—follow along!

1. Prerequisites

• Node.js: 18+ (node --version).
• npm: 8+ (npm --version).
• Hyperbrowser API Key: Sign up at hyperbrowser.ai for a free tier key.

• Browser: Chrome or Firefox for Developer Tools.
• Disk Space: ~500MB for Hyperbrowser app and dependencies.

2. Install Hyperbrowser’s DeepCrawler

Clone the example repository:

git clone https://github.com/hyperbrowserai/hyperbrowser-app-examples.git
cd hyperbrowser-app-examples
cd deep-crawler-bot

Install dependencies:

npm install

Set up environment variables:

cp .env.example .env.local

Edit .env.local with your Hyperbrowser API key:

HYPERBROWSER_API_KEY=your_api_key_here

Run the development server:

npm run dev

Open your browser and navigate to http://localhost:3000. You’ll see the Hyperbrowser DeepCrawler interface, ready for API scanning!

3. Find APIs with Hyperbrowser’s DeepCrawler

In the Hyperbrowser UI (http://localhost:3000):

• Enter the target URL (e.g., https://retouched.ai or https://github.com).
• Enable options: Use Proxy and Solve CAPTCHAs.
• Click Start Crawl to scan for API endpoints.
  Example output (in the UI or exported JSON):

{
  "endpoints": [
    {
      "url": "https://api.retouched.ai/v1/background-removal",
      "method": "POST",
      "headers": { "Content-Type": "application/json" },
      "description": "Handles image background removal"
    }
  ]
}

Hyperbrowser scanned retouched.ai and found the background removal API in ~50 seconds! Export to Postman via the dashboard (app.hyperbrowser.ai).

4. Find APIs Without AI (Developer Tools)

For a non-AI approach, use browser Developer Tools:

• Open Chrome/Firefox, navigate to https://retouched.ai.
• Press F12 to open Developer Tools > Network tab > XHR filter.

• Interact with the site (e.g., upload an image for background removal).
• Look for API calls (e.g., https://api.retouched.ai/v1/background-removal).
• Example call:

{
  "method": "POST",
  "url": "https://api.retouched.ai/v1/background-removal",
  "body": { "image": "base64-encoded-image" }
}

• Right-click the request, copy as curl:

curl -X POST https://api.retouched.ai/v1/background-removal \
  -H "Content-Type: application/json" \
  -d '{"image":"base64-encoded-image"}'

This took me ~5 minutes of clicking around—way slower than Hyperbrowser!

5. Test the Discovered API

Test the API with a curl request (use an API key if required):

curl -X POST https://api.retouched.ai/v1/background-removal \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{"image":"data:image/jpeg;base64,/9j/..."}'

Or use the Hyperbrowser UI:

• Go to the Test tab in http://localhost:3000.
• Enter the endpoint URL and parameters (e.g., image data).
• Click Run Test. Hyperbrowser handles CAPTCHAs and returns results, like processed image data.
  I tested the background removal API, and it worked like a charm!

You can also use tools like Apidog

or Python’s requests library to send requests and verify the API behavior.

import requests

url = 'https://example.com/api/data'
headers = {'Authorization': 'Bearer your_token_here'}

response = requests.get(url, headers=headers)
print(response.json())

6. Troubleshoot Common Issues

• Hyperbrowser Errors:
• API Key Invalid: Verify key in app.hyperbrowser.ai.
• Server Not Running: Check http://localhost:3000. Restart with npm run dev.
• Task Timeout: Increase timeoutMinutes in the UI settings.
• Developer Tools Issues:
• No XHR Requests: Try the Fetch filter or interact more with the site.
• CORS Errors: Use a proxy or test in Postman.
• General: Ensure Node.js 18+ and stable internet. See docs.hyperbrowser.ai or X for tips.

Customizing and Extending Hyperbrowser

Level up your API hunting:

• Export Formats: Save endpoints as OpenAPI YAML via Hyperbrowser dashboard.
• LangChain Integration: Use HyperbrowserLoader for structured data:

const { HyperbrowserLoader } = require('langchain_hyperbrowser');
const loader = new HyperbrowserLoader({ urls: 'https://retouched.ai' });
loader.load().then(docs => console.log(docs[0]));

• Contribute: Submit PRs to github.com/hyperbrowserai.

I exported my APIs to Apidog and felt like an API pro!

Why Hyperbrowser Beats Manual API Discovery

Hyperbrowser’s AI-driven approach outshines manual Developer Tools for speed and ease, especially on complex sites with anti-bot protections. Unlike Google Analytics’ tracking APIs, Hyperbrowser focuses on actionable endpoint discovery for developers. The only catch? You need a Hyperbrowser account, but the free tier is solid. The Hyperbrowser docs and community are gold for support.

Ready to uncover APIs with Hyperbrowser? Fire up that deep-crawler-bot and scan a site—I’m stoked to see your API wins!