SERP Scraping for Paperclip AI Agents: Real Search Data at Scale

Paperclip is an open-source orchestration layer for autonomous AI companies. You define a company goal, hire agents under an org chart, and Paperclip handles scheduling, delegation, cost tracking, and governance. What Paperclip doesn't include by default is live data from the open web - and that's where the Autom API fits.

This article shows how to wire Autom's SERP endpoints into Paperclip agents, with concrete examples for the workflows your agents are most likely to run.

See also: SERP Scraping with OpenClaw for a no-code alternative using HTTP Action nodes.

Why agents need live search data

An agent working toward a goal like "reach the top 5 for target keywords" or "monitor competitor content daily" cannot operate on stale data. Static knowledge cutoffs and cached datasets produce plans that don't reflect reality.

Live SERP data gives agents:

• Current ranking positions across Google, Bing, and Brave - not snapshots from months ago
• Competitor movements detected in real time, not surfaced in a weekly report
• Market signals from news, shopping, and video results that inform content strategy
• Freshness awareness - what's trending in the index today, not last month

For a Paperclip company whose goal is growth or market intelligence, SERP data is a first-class input, not an optional enrichment.

How Paperclip agents make HTTP calls

Paperclip agents interact with external services through tool calls. Any agent that can run HTTP requests - OpenClaw, Claude, Codex, a custom Bash agent - can call the Autom API directly.

The recommended pattern is to teach your agents the API surface using a SKILLS.md file. Paperclip injects skill files at runtime so agents know what tools are available without retraining.

SKILLS.md: teaching agents to use Autom

Create a SKILLS.md in your Paperclip project with the following section:

## Autom SERP API

Fetch live search engine results using the Autom API. Use this to check rankings,
monitor competitors, or gather content signals.

Authentication: set the `x-api-key` header with the value from the `AUTOM_API_KEY` env var.

### Google Search
POST https://api.autom.dev/v1/google/search
{
  "query": "<your query>",
  "page": 1,
  "gl": "us",
  "hl": "en"
}
Returns: organic_results[], pagination, search_parameters

### Google News
POST https://api.autom.dev/v1/google/news
{ "query": "<topic>", "gl": "us", "hl": "en" }
Returns: organic_results[] with date, source, snippet

### Bing Search
POST https://api.autom.dev/v1/bing/search
{ "query": "<your query>", "page": 1 }
Returns: organic_results[], search_parameters

### Brave Search
POST https://api.autom.dev/v1/brave/search
{ "query": "<your query>", "page": 0 }
Returns: organic_results[], search_parameters

All responses use the same top-level schema. Bing and Google share identical field names.
Brave uses zero-based pagination (page: 0 for first page).
Cost: 1 credit per call. Only successful 200 responses are billed.

With this file in place, any agent that reads SKILLS.md - which Paperclip handles automatically - knows how to query all three engines without further configuration.

Setting up the API key

Generate an Autom API key at app.autom.dev and store it as an environment variable in your Paperclip deployment:

AUTOM_API_KEY=your_key_here

Reference it as {{AUTOM_API_KEY}} in your SKILLS.md and agent configurations. Never hardcode the key in a prompt or config file.

Heartbeat-driven rank tracking

Paperclip's heartbeat system schedules agents to wake up on a fixed interval, check their work, and act. This maps directly to recurring SERP monitoring.

Example: SEO Analyst agent running every 8 hours

agent: SEO Analyst
runtime: claude
heartbeat: every 8h
goal: Track keyword rankings and surface position changes > 2 spots
skills:
  - SKILLS.md

On each heartbeat, the agent:

1. Reads its keyword list from the ticket or project context
2. Calls POST /v1/google/search for each keyword with gl and hl matching the target market
3. Compares positions against the last stored result
4. Creates a ticket for the SEO Manager if a significant drop is detected
5. Logs results to the project's audit trail

The cost per run is 1 credit per keyword per engine. For a 20-keyword list querying Google and Bing, that's 40 credits every 8 hours - predictable and controllable with Paperclip's per-agent budget system.

Goal-aligned SERP analysis

Paperclip propagates company goals down the org chart. Every task an agent runs carries context that traces back to the company mission. A task like:

Company goal: Reach $1M ARR with the #1 AI note-taking app
Project goal: Outrank competitors for "AI note-taking" queries
Agent goal:   Pull weekly ranking data for 15 target keywords

…ensures the agent knows why it's pulling SERP data, not just what to pull. This matters when the agent needs to decide which signals to flag, which competitors to watch, and how to prioritize its own tool calls.

The Autom response gives agents the raw material to reason about that context:

{
    "organic_results": [
        {
            "position": 1,
            "title": "Notion AI - Smart Notes & Docs",
            "link": "https://notion.so",
            "domain": "notion.so",
            "snippet": "..."
        },
        {
            "position": 4,
            "title": "Your App Name",
            "link": "https://yourapp.com",
            "domain": "yourapp.com",
            "snippet": "..."
        }
    ],
    "search_parameters": {
        "q": "AI note-taking app",
        "gl": "us",
        "hl": "en",
        "page": 1,
        "engine": "google"
    }
}

The agent sees position 4, compares it to the goal (top 1), and knows to escalate - without being told to check.

Org chart use cases

Paperclip's org chart lets you assign different SERP tasks to different roles. Here are common patterns:

Content team

CMO
 └── SEO Analyst (heartbeat: 8h)   → rank tracking via Google Search
 └── Content Writer (heartbeat: 4h) → topic research via Google News + Autocomplete
 └── Social Manager (heartbeat: 12h)→ trend signals via Google Trends / News

The Content Writer agent uses Google News to find what's being published in the niche before drafting:

POST https://api.autom.dev/v1/google/news
Content-Type: application/json
x-api-key: {{AUTOM_API_KEY}}

{
    "query": "AI productivity tools 2026",
    "gl": "us",
    "hl": "en"
}

The response surfaces what publishers are covering, which angles are getting traction, and which sources dominate. The agent uses this to brief the article outline before writing - no manual research required.

Market intelligence team

CEO
 └── Research Analyst (heartbeat: 24h)
       → competitor presence via Google + Bing + Brave (parallel)
       → shopping data via Google Shopping for product-intent queries
       → job listings via Google Jobs to infer competitor hiring signals

Running all three engines in parallel on the same query gives the Research Analyst a cross-index view. When a competitor appears in the top 5 on all three engines for a target keyword, that's a stronger signal than a single-engine hit.

Lead generation team

Head of Sales
 └── Prospector (heartbeat: 24h)
       → Google Search with location parameter to find businesses by niche + city
       → Google Maps for local business data (experimental)

POST https://api.autom.dev/v1/google/search
Content-Type: application/json
x-api-key: {{AUTOM_API_KEY}}

{
    "query": "boutique coffee roaster",
    "gl": "us",
    "location": "Austin, Texas"
}

The Prospector extracts domains from organic_results, deduplicates against the existing CRM list, and creates outreach tickets for new leads.

Cost control per agent

Every Autom call costs 1 credit. Paperclip's budget system enforces a hard cap per agent per month. When an agent hits its budget, it stops - no runaway loops, no surprise bills.

A practical budget for a 5-agent SERP team:

Paperclip surfaces cost per agent, per task, and per project - so you can see which agents are expensive and which queries burn the most tokens before the budget is gone.

Ticket-triggered SERP queries

Beyond heartbeats, agents respond to tickets. Any team member - human or agent - can open a ticket that triggers a SERP query on demand:

Ticket #1043
"Check where we rank for 'open source task manager' on Bing and Google"
Assigned to: SEO Analyst

The SEO Analyst wakes, calls both endpoints, and posts the results back to the ticket thread with full tool-call tracing. The audit log shows exactly which API calls were made, what they returned, and how the agent used the data.

Error handling

Handle Autom API errors gracefully in your agent instructions or SKILLS.md:

• 429 – rate limit: wait and retry with exponential backoff (start at 2 s, cap at 30 s)
• 500 – internal error: retry up to 3 times before escalating to a manager agent via ticket
• Empty results: if organic_results.length === 0, log the miss and skip downstream processing - don't create noise in the ticket system
• Only 200 responses are billed: failed requests cost nothing, so retries are always safe

What Autom handles so your agents don't have to

SERP scraping from scratch requires rotating proxies, CAPTCHA solving, HTML parsing, and constant maintenance as search engines update their markup. None of that is a good use of agent compute.

Autom abstracts the entire stack - proxy rotation, bot detection evasion, HTML parsing, structured JSON delivery - and returns clean, consistent data your agents can reason about directly.

Every endpoint shares the same top-level schema (organic_results, pagination, search_parameters), which means a single parsing step in your SKILLS.md covers all engines and content types.

Full API surface for your agents

All endpoints use the same x-api-key header and cost 1 credit per call:

A Paperclip company can chain these endpoints across its org chart: the SEO Analyst tracks Google rankings, the Research Analyst cross-references with Bing and Brave, the Content Writer sources ideas from Google News, and the Prospector finds leads via Google Maps - all running autonomously, all within budget.

Get your API key at app.autom.dev and browse the full reference at docs.autom.dev. Star Paperclip on GitHub to follow the project.