Automation & Workflow

Google Search Scraper with n8n

Use Google Search Scraper with n8n with Autom

Build powerful Google search scrapers using n8n workflow automation platform and Autom.dev API. Extract SERP data, automate SEO research, and create intelligent search workflows without coding.

Quick Start with Autom.dev API

Example: Google Search Scraping

Use the Autom.dev API directly in your n8n workflows to scrape Google search results:

POST https://autom.dev/api/v1/google/search
Content-Type: application/json

{
    "query": "google search",
    "page": 1,
    "gl": "us",
    "hl": "en",
    "num": 10
}

Response Example:

{
    "pagination": {
        "next": 2,
        "current": 1,
        "other_pages": [
            2,
            3,
            4,
            5,
            6,
            7,
            8,
            9,
            10
        ],
        "has_next_page": true
    },
    "organic_results": [
        {
            "link": "https://developers.google.com/custom-search/v1/overview",
            "title": "Google for Developers",
            "domain": "developers.google.com",
            "source": "Google for Developers",
            "snippet": "Use REST to Invoke the API",
            "position": 1,
            "displayed_link": "https://developers.google.com › overview",
            "snippet_matched": [
                "api"
            ]
        },
        {
            "link": "https://support.google.com/websearch/thread/135474043/how-do-i-get-web-search-results-using-an-api?hl=en",
            "title": "Google Help",
            "domain": "support.google.com",
            "source": "Google Help",
            "snippet": "Nov 15, 2021 —",
            "position": 2,
            "displayed_link": "https://support.google.com › websearch › thread › how-...",
            "snippet_matched": []
        },
        {
            "link": "https://console.cloud.google.com/marketplace/product/google/customsearch.googleapis.com",
            "title": "Google Cloud console",
            "domain": "console.cloud.google.com",
            "source": "Google Cloud console",
            "snippet": "The JSON Custom Search API lets you develop websites and applications to retrieve and display search results from Google Custom Search programmatically.",
            "position": 3,
            "displayed_link": "https://console.cloud.google.com › product › google › c...",
            "snippet_matched": [
                "search",
                "google",
                "api"
            ]
        },
        {
            "link": "https://www.searchapi.io/",
            "title": "Google Search API for real-time SERP scraping",
            "domain": "www.searchapi.io",
            "source": "SearchApi",
            "snippet": "Scrape organic results, ads, related searches, inline questions and more. Receive structured results in JSON format. Sign up for a free plan today!",
            "position": 4,
            "displayed_link": "https://www.searchapi.io",
            "snippet_matched": [
                "search"
            ]
        },
        {
            "link": "https://serpapi.com/",
            "title": "SerpApi: Google Search API",
            "domain": "serpapi.com",
            "source": "SerpApi",
            "snippet": "SerpApi is a real-time API to access Google search results. We handle proxies, solve captchas, and parse all rich structured data for you.",
            "position": 5,
            "displayed_link": "https://serpapi.com",
            "snippet_matched": [
                "search",
                "google",
                "api"
            ]
        },
        {
            "link": "https://medium.com/@TerryFrederick/google-search-api-everything-you-need-to-know-fd1e24cfca45",
            "title": "Medium · Terry Frederick",
            "domain": "medium.com",
            "source": "Medium · Terry Frederick",
            "snippet": "We refer to the Google Search API as an interface that facilitates the interaction between developers or applications and Google's search infrastructure.",
            "position": 6,
            "displayed_link": "30+ likes · 1 year ago",
            "snippet_matched": [
                "search",
                "google",
                "api"
            ]
        },
        {
            "link": "https://developers.google.com/webmaster-tools",
            "title": "Google for Developers",
            "domain": "developers.google.com",
            "source": "Google for Developers",
            "snippet": "The Search Console API provides programmatic access to the most popular reports and actions in your Search Console account.",
            "position": 7,
            "displayed_link": "https://developers.google.com › webmaster-tools",
            "snippet_matched": [
                "search",
                "api"
            ]
        },
        {
            "link": "https://ai.google.dev/gemini-api/docs/google-search",
            "title": "Google AI for Developers",
            "domain": "ai.google.dev",
            "source": "Google AI for Developers",
            "snippet": "Grounding with Google Search connects the Gemini model to real-time web content and works with all available languages. This allows Gemini to provide more ...",
            "position": 8,
            "displayed_link": "https://ai.google.dev › gemini-api › docs › google-search",
            "snippet_matched": [
                "search",
                "google"
            ]
        },
        {
            "link": "https://cloud.google.com/docs/authentication/api-keys",
            "title": "Google Cloud",
            "domain": "cloud.google.com",
            "source": "Google Cloud",
            "snippet": "This page describes how to create, edit, and restrict API keys. For information about how to use API keys to access Google APIs, see Use API keys to access ...",
            "position": 9,
            "displayed_link": "https://cloud.google.com › Cloud SDK › Authentication",
            "snippet_matched": [
                "google",
                "api"
            ]
        },
        {
            "link": "https://www.youtube.com/watch?v=4YhxXRPKI0c",
            "title": "YouTube · Jie Jenn",
            "domain": "www.youtube.com",
            "source": "YouTube · Jie Jenn",
            "snippet": "YouTube · Jie Jenn12.6K+ views · 1 year ago",
            "position": 10,
            "displayed_link": "12.6K+ views · 1 year ago",
            "snippet_matched": []
        }
    ],
    "search_parameters": {
        "q": "google search api",
        "gl": "us",
        "hl": "en",
        "num": 10,
        "page": 1,
        "uule": null,
        "engine": "google",
        "location": null
    },
    "search_information": {
        "total_results": 4410000000,
        "time_taken_displayed": 0.28
    }
}

Key Features

Visual Workflow Builder: Drag-and-drop interface for creating search automation
Real-Time SERP Data: Access live Google search results instantly
1000+ Integrations: Connect search data with any app or service
AI-Powered Workflows: Enhance AI agents with current search data
Auto-Scheduled Monitoring: Track rankings and competitors automatically

Perfect For

SEO Monitoring: Automated rank tracking and competitive analysis
Market Research: Extract product data and pricing information
Content Strategy: Identify trending topics and search patterns
AI Enhancement: Feed real-time search data to AI models
Lead Generation: Extract business information from search results

Supported Search Types

Google Search: Web results, images, shopping, news
Local SEO: Google Maps and local business data
E-commerce: Product prices and availability
News Monitoring: Latest articles and trending topics
Academic Research: Google Scholar integration

n8n Workflow Example

1. Create HTTP Request Node

Method: POST
URL: https://autom.dev/api/v1/google/search
Headers: x-api-key: YOUR_API_KEY
Body: JSON with search parameters

2. Process Results

Extract organic results
Filter by position or domain
Transform data format
Store in database or send to other apps

3. Automation Triggers

Schedule: Daily/weekly rank monitoring
Webhook: Real-time search triggers
Manual: On-demand research

Common Use Cases

SEO Rank Tracking

Monitor keyword positions automatically and get alerts for ranking changes.

Competitive Analysis

Track competitor rankings and content strategies across multiple keywords.

Content Research

Identify trending topics and content gaps in your niche.

Price Monitoring

Track product prices and availability across e-commerce platforms.

Lead Generation

Extract business contact information from search results.

Getting Started

Sign up for Autom.dev: Get your API key from autom.dev
Install n8n: Use Docker, npm, or n8n Cloud
Create Workflow: Add HTTP Request node with Autom.dev endpoint
Configure Search: Set your query parameters and filters
Automate: Schedule or trigger your search workflows

API Parameters

query: Search term or keyword
page: Results page number (1-10)
gl: Country code (us, uk, fr, etc.)
hl: Language code (en, fr, es, etc.)
num: Number of results (1-100)
device: desktop or mobile
location: Specific location for local search

Enterprise Features

Rate Limiting: Automatic request throttling
Error Handling: Retry mechanisms and fallbacks
Data Export: CSV, JSON, or database integration
Team Collaboration: Share workflows and templates
Monitoring: Real-time execution tracking

Start building your Google search scraper today with n8n and Autom.dev - no coding required!

Complete n8n HTTP Request Configuration

Based on the n8n HTTP Request documentation, here's how to properly configure the node for Autom.dev API:

Visual Configuration Guide

n8n HTTP Request Configuration

The image above shows the exact configuration needed in n8n. Follow these 3 key steps:

Step 1: Configure the API URL 🌐

URL Field: Enter https://autom.dev/api/v1/google/search

Set Method to POST
This is the Autom.dev endpoint for Google Search scraping

Step 2: Set up Authentication Header 🔑

Header Configuration:

Name: x-api-key
Value: Your Autom.dev API token
Make sure Send Headers is enabled (toggle ON)

Step 3: Add Search Parameters 📋

Body Configuration:

Enable Send Body (toggle ON)
Set Body Content Type to JSON
Add your search payload:

{
    "query": "google search api",
    "page": 1,
    "gl": "us",
    "hl": "en",
    "num": 10
}

Step-by-Step Configuration

1. Basic Request Settings

Add HTTP Request Node: Drag the HTTP Request node to your workflow canvas
Method: Select POST from the dropdown
URL: Enter https://autom.dev/api/v1/google/search

2. Authentication Header Configuration

Go to Headers section: Click on "Add Parameter" under Headers
Header Name: Enter x-api-key
Header Value: Enter your Autom.dev API key (get it from autom.dev)

Name: x-api-key
Value: your_actual_api_key_here

3. Request Body Configuration

Body Content Type: Select JSON
JSON Body: Add your search parameters

{
    "query": "google search api",
    "page": 1,
    "gl": "us",
    "hl": "en",
    "num": 10
}

4. Additional HTTP Request Options

Response Handling:

Include Response Headers and Status: Turn ON to get full response details
Response Format: Keep as "Autodetect" for JSON responses

Error Handling:

Never Error: Turn OFF to catch API errors properly
Ignore SSL Issues: Keep OFF for security

Timeout Settings:

Timeout: Set to 30000ms (30 seconds) for search requests

Example n8n Workflow Structure

[Manual Trigger] → [HTTP Request (Autom.dev)] → [Process Results] → [Output]

Processing Autom.dev Response

After receiving the API response, you can extract data using n8n's built-in functions:

Extract Organic Results

// In a Code node or Function node
const results = $json.organic_results;
return results.map(result => ({
    position: result.position,
    title: result.title,
    link: result.link,
    snippet: result.snippet,
    domain: result.domain
}));

Filter by Position

// Get only top 5 results
const topResults = $json.organic_results
    .filter(result => result.position <= 5);
return topResults;

Extract Specific Domains

// Filter results from specific domains
const filteredResults = $json.organic_results
    .filter(result => result.domain.includes('wikipedia.org'));
return filteredResults;

Get Started with n8n