Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
e2a7544
feat: n8n user docs for WCC actor app
protoss70 Aug 5, 2025
8c05fc4
fix: image path update
protoss70 Aug 5, 2025
2a6822a
fix: minor md fix
protoss70 Aug 5, 2025
41701e0
fix: doc image path update
protoss70 Aug 5, 2025
090a292
feat: improve docs for WCC n8n app
protoss70 Aug 11, 2025
5bbe301
Merge remote-tracking branch 'origin/master' into feat/n8n-wcc-docs
protoss70 Aug 11, 2025
7bdb83a
fix: minor image path fix
protoss70 Aug 11, 2025
028d4bc
feat: ai agent use case docs
protoss70 Aug 18, 2025
313cebd
fix: remove repetition
protoss70 Aug 19, 2025
7c7aec9
feat: improve wording
protoss70 Aug 19, 2025
59ee9a3
feat: improvements
protoss70 Aug 19, 2025
646cf47
Update sources/platform/integrations/workflows-and-notifications/n8n/…
protoss70 Aug 25, 2025
7a9ce7e
Merge remote-tracking branch 'origin' into feat/n8n-wcc-docs
protoss70 Sep 10, 2025
4cb4646
feat: update docs according to n8n WCC changes
protoss70 Sep 10, 2025
d7d937a
fix: md fix
protoss70 Sep 10, 2025
4ae35e8
fix: minor fixes
protoss70 Sep 11, 2025
dc1ff9c
Merge remote-tracking branch 'origin/master' into feat/n8n-wcc-docs
protoss70 Sep 11, 2025
20a90e6
fix: Update sources/platform/integrations/workflows-and-notifications…
protoss70 Sep 22, 2025
afa76fe
fix: Update sources/platform/integrations/workflows-and-notifications…
protoss70 Sep 22, 2025
6c0884b
Merge remote-tracking branch 'origin/master' into feat/n8n-wcc-docs
protoss70 Sep 22, 2025
8c9f2f0
fix: minor wording fix
protoss70 Sep 22, 2025
d525846
feat: doc re-ordering
protoss70 Sep 22, 2025
5bff6af
fix: linting fix
protoss70 Sep 22, 2025
62133cb
fix: vale fix
protoss70 Sep 22, 2025
6654136
fix: wording fix
protoss70 Sep 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ If you're running a self-hosted n8n instance, you can install the Apify communit
1. Agree to the [risks](https://docs.n8n.io/integrations/community-nodes/risks/) of using community nodes and select **Install**.
1. You can now use the node in your workflows.

![Apify Install Node](../images/n8n-install-node-self-hosted.png)
![Apify Install Node](../../images/n8n-install-node-self-hosted.png)

## Install the Apify Node (n8n Cloud)

Expand All @@ -42,7 +42,7 @@ For n8n Cloud users, installation is even simpler and doesn't require manual pac
1. Search for **Apify** in the community node registry
1. Click **Install node** to add the Apify node to your instance

![Apify Install Node](../images/n8n-install-node-cloud.png)
![Apify Install Node](../../images/n8n-install-node-cloud.png)

:::note Verified community nodes visibility

Expand All @@ -63,7 +63,7 @@ The Apify node offers two authentication methods to securely connect to your Api
1. Enter your Apify API token. (find it in the [Apify Console](https://console.apify.com/settings/integrations)).
1. Click **Save**.

![Apify Auth](../images/n8n-api-auth.png)
![Apify Auth](../../images/n8n-api-auth.png)

### OAuth2 (cloud instance only)

Expand All @@ -72,8 +72,8 @@ The Apify node offers two authentication methods to securely connect to your Api
1. Select **Connect my account** and authorize with your Apify account.
1. n8n automatically retrieves and stores the OAuth2 tokens.

![Apify Auth](../images/n8n-oauth.png)

![Apify Auth](../../images/n8n-oauth.png)
:::note

For simplicity on n8n Cloud, use the API key method if you prefer manual control over credentials.
Expand All @@ -92,7 +92,7 @@ Start by building a basic workflow in n8n, then add the Apify node to handle tas
1. In the node's **Credentials** dropdown, choose the Apify credential you configured earlier. If you haven't configured any credentials, you can do so in this step. The process will be the same.
1. You can now use Apify node as a trigger or action in your workflow.

![Apify Node](../images/n8n-list-of-operations.png)
![Apify Node](../../images/n8n-list-of-operations.png)

## Use Apify node as trigger

Expand All @@ -107,7 +107,7 @@ Triggers let your workflow respond automatically to events in Apify, such as whe
1. Add subsequent nodes (e.g., HTTP Request, Google Sheets) to process or store the output.
1. Save and execute the workflow.

![Apify Node](../images/n8n-trigger-example.png)
![Apify Node](../../images/n8n-trigger-example.png)

## Use Apify node as an action

Expand All @@ -122,13 +122,13 @@ Actions allow you to perform operations like running an Actor within a workflow.
- **Memory**: Amount of memory allocated for the Actor run, in megabytes
- **Build Tag**: Specifies the Actor build tag to run. By default, the run uses the build specified in the default run configuration for the Actor (typically `latest`)
- **Wait for finish**: Whether to wait for the run to finish before continuing. If true, the node will wait for the run to complete (successfully or not) before moving to the next node
![Apify Node](../images/n8n-run-actor-example.png)
![Apify Node](../../images/n8n-run-actor-example.png)
1. Add another Apify operation called **Get Dataset Items**.
- Set **Dataset ID** parameter as **defaultDatasetId** value received from the previous **Run Actor** node. This will give you the output of the Actor run
![Apify Node](../images/n8n-get-dataset-items-example.png)
![Apify Node](../../images/n8n-get-dataset-items-example.png)
1. Add any subsequent nodes (e.g. Google Sheets) to process or store the output
1. Save and execute the workflow
![Apify Node](../images/n8n-workflow-example.png)
![Apify Node](../../images/n8n-workflow-example.png)

## Available Operations

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
---
title: n8n - Website Content Crawler by Apify
description: Learn about Website Content Crawler module.
sidebar_label: Website Content Crawler
sidebar_position: 6
slug: /integrations/n8n/website-content-crawler
toc_max_heading_level: 4
---

Website Content Crawler from [Apify](https://apify.com/apify/website-content-crawler) lets you extract text content from websites to feed AI models, LLM applications, vector databases, or Retrieval Augmented Generation (RAG) pipelines. It supports rich formatting using Markdown, cleans the HTML of irrelevant elements, downloads linked files, and integrates with AI ecosystems like Langchain, LlamaIndex, and other LLM frameworks.

To use these modules, you need an [API token](https://docs.apify.com/platform/integrations/api#api-token). You can find your token in the [Apify Console](https://console.apify.com/) under **Settings > Integrations**. After connecting, you can automate content extraction at scale and incorporate the results into your AI workflows.

## Prerequisites

Before you begin, make sure you have:

- An [Apify account](https://console.apify.com/)
- An [n8n instance](https://docs.n8n.io/getting-started/) (self‑hosted or cloud)

## n8n Cloud setup

This section explains how to install and connect the Apify node when using n8n Cloud.

### Install

For n8n Cloud users, installation is even simpler and doesn't require manual package entry. Just search and add the node from the canvas.

1. Go to the **Canvas** and open the **nodes panel**
1. Search for **Website Content Crawler by Apify** in the community node registry
1. Click **Install node** to add the Apify node to your instance

![Website Content Crawler by Apify on n8n](images/operations.png)

:::note Verified community nodes visibility

On n8n Cloud, instance owners can toggle visibility of verified community nodes in the Cloud Admin Panel. Ensure this setting is enabled to install the Website Content Crawler by Apify node.

:::

### Connect

1. In n8n Cloud, select **Create Credential**.
1. Search for Apify OAuth2 API and select **Continue**.
1. Select **Connect my account** and authorize with your Apify account.
1. n8n automatically retrieves and stores the OAuth2 tokens.

![Apify Auth](images/credentials.png)

:::note Cloud API Key management

On n8n Cloud, you can use the API key method if you prefer to manage your credentials manually.
See the [**Connect** section for n8n self-hosted](#connect-self-hosted) for detailed API configuration instructions.

:::

With authentication set up, you can now create workflows that incorporate the Apify node.


## n8n self-hosted setup

This section explains how to install and connect the Apify node when running your own n8n instance.

### Install

If you're running a self-hosted n8n instance, you can install the Apify community node directly from the editor. This process adds the node to your available tools, enabling Apify operations in workflows.

1. Open your n8n instance.
1. Go to **Settings > Community Nodes**.
1. Select **Install**.
1. Enter the npm package name: `@apify/n8n-nodes-apify-content-crawler` (for latest version). To install a specific [version](https://www.npmjs.com/package/@apify/n8n-nodes-apify-content-crawler?activeTab=versions) enter e.g `@apify/[email protected]`.
1. Agree to the [risks](https://docs.n8n.io/integrations/community-nodes/risks/) of using community nodes and select **Install**.
1. You can now use the node in your workflows.

![Apify Install Node](images/install.png)

<a id="connect-self-hosted"></a>

### Connect

1. Create an account at [Apify](https://console.apify.com/). You can sign up using your email, Gmail, or GitHub account.

![Sign up page](../make/images/ai-crawling/wcc-signup.png)

1. To connect your Apify account to n8n, you can use an OAuth connection (recommended) or an Apify API token. To get the Apify API token, navigate to **[Settings > API & Integrations](https://console.apify.com/settings/integrations)** in the Apify Console.

![Apify Console token for n8n](../make/images/Apify_Console_token_for_Make.png)

1. Find your token under **Personal API tokens** section. You can also create a new API token with multiple customizable permissions by clicking on **+ Create a new token**.
1. Click the **Copy** icon next to your API token to copy it to your clipboard. Then, return to your n8n workflow interface.

![Apify token on n8n](../make/images/Apify_token_on_Make.png)

1. In n8n, click **Create new credential** of the chosen Apify Scraper module.
1. In the **API key** field, paste the API token you copied from Apify and click **Save**.

![Apify token on n8n](images/token.png)


## Website Content Crawler by Apify module

This module provides complete control over the content extraction process, allowing you to fine-tune every aspect of the crawling and transformation pipeline. This module is ideal for complex websites, JavaScript-heavy applications, or when you need precise control over content extraction.

### Key features

- _Multiple Crawler Options_: Choose between headless browsers (Playwright) or faster HTTP clients (Cheerio)
- _Custom Content Selection_: Specify exactly which elements to keep or remove
- _Advanced Navigation Control_: Set crawling depth, scope, and URL patterns
- _Dynamic Content Handling_: Wait for JavaScript-rendered content to load
- _Interactive Element Support_: Click expandable sections to reveal hidden content
- _Multiple Output Formats_: Save content as Markdown, HTML, or plain text
- _Proxy Configuration_: Use proxies to handle geo-restrictions or avoid IP blocks
- _Content Transformation Options_: Multiple algorithms for optimal content extraction

### How it works

the Website Content Crawler by Apify module provides granular control over the entire crawling process. For _Crawler selection_, you can choose from Playwright (Firefox/Chrome) or Cheerio, depending on the complexity of the target website. _URL management_ allows you to define the crawling scope with include and exclude URL patterns. You can also exercise precise _DOM manipulation_ by controlling which HTML elements to keep or remove. To ensure the best results, you can apply specialized algorithms for _Content transformation_ and select from various _Output formatting_ options for better AI model compatibility.

### Output data

For each crawled web page, you'll receive:

- _Page metadata_: URL, title, description, canonical URL
- _Cleaned text content_: The main article content with irrelevant elements removed
- _Markdown formatting_: Structured content with headers, lists, links, and other formatting preserved
- _Crawl information_: Loaded URL, referrer URL, timestamp, HTTP status
- _Optional file downloads_: PDFs, DOCs, and other linked documents
- _Multiple format options_: Content in Markdown, HTML, or plain text
- _Debug information_: Detailed extraction diagnostics and snapshots
- _HTML transformations_: Results from different content extraction algorithms
- _File storage options_: Flexible storage for HTML, screenshots, or downloaded files

```json title="Sample output (shortened)"
{
"url": "https://docs.apify.com/academy/web-scraping-for-beginners",
"crawl": {
"loadedUrl": "https://docs.apify.com/academy/web-scraping-for-beginners",
"loadedTime": "2025-04-22T14:33:20.514Z",
"referrerUrl": "https://docs.apify.com/academy",
"depth": 1,
"httpStatusCode": 200
},
"metadata": {
"canonicalUrl": "https://docs.apify.com/academy/web-scraping-for-beginners",
"title": "Web scraping for beginners | Apify Documentation",
"description": "Learn the basics of web scraping with a step-by-step tutorial and practical exercises.",
"languageCode": "en",
"markdown": "# Web scraping for beginners\n\nWelcome to our comprehensive web scraping tutorial for beginners. This guide will take you through the fundamentals of extracting data from websites, with practical examples and exercises.\n\n## What is web scraping?\n\nWeb scraping is the process of extracting data from websites. It involves making HTTP requests to web servers, downloading HTML pages, and parsing them to extract the desired information.\n\n## Why learn web scraping?\n\n- **Data collection**: Gather information for research, analysis, or business intelligence\n- **Automation**: Save time by automating repetitive data collection tasks\n- **Integration**: Connect web data with your applications or databases\n- **Monitoring**: Track changes on websites automatically\n\n## Getting started\n\nTo begin web scraping, you'll need to understand the basics of HTML, CSS selectors, and HTTP. This tutorial will guide you through these concepts step by step.\n\n...",
"text": "Web scraping for beginners\n\nWelcome to our comprehensive web scraping tutorial for beginners. This guide will take you through the fundamentals of extracting data from websites, with practical examples and exercises.\n\nWhat is web scraping?\n\nWeb scraping is the process of extracting data from websites. It involves making HTTP requests to web servers, downloading HTML pages, and parsing them to extract the desired information.\n\nWhy learn web scraping?\n\n- Data collection: Gather information for research, analysis, or business intelligence\n- Automation: Save time by automating repetitive data collection tasks\n- Integration: Connect web data with your applications or databases\n- Monitoring: Track changes on websites automatically\n\nGetting started\n\nTo begin web scraping, you'll need to understand the basics of HTML, CSS selectors, and HTTP. This tutorial will guide you through these concepts step by step.\n\n..."
}
}
```

You can access any of thousands of our scrapers on Apify Store by using the [general Apify app](https://n8n.io/integrations/apify).

### Configuration options

You can select the _Crawler type_ by choosing the rendering engine (browser or HTTP client) and the _Content extraction algorithm_ from multiple HTML transformers. _Element selectors_ allow you to specify which elements to keep, remove, or click, while _URL patterns_ let you define inclusion and exclusion rules with glob syntax. You can also set _Crawling parameters_ like concurrency, depth, timeouts, and retries. For robust crawling, you can configure _Proxy configuration_ settings and select from various _Output options_ for content formats and storage.

## Usage as an AI Agent Tool

You can setup Apify's Scraper for AI Crawling node as a tool for your AI Agents.

![Setup AI Agent](./images/setup.png)

### Dynamic URL crawling

In the Website Content Crawler module you can set the **Start URLs** to be filled in by your AI Agent dynamically. This allows the Agent to decide on which pages to scrape off the internet.

Two key parameters to configure for optimized AI Agent usage are **Max crawling depth** and **Max pages**. Remember that the scraping results are passed into the AI Agent’s context, so using smaller values helps stay within context limits.

![Apify Configuration](./images/config.png)

### Example usage

Here, the agent was used to find information about Apify's latest blog post. It correctly filled in the URL for the blog and summarized its content.

![Scraping Results](./images/result.png)