FireCrawl

FireCrawl Document Loader

FireCrawl is a powerful web crawling and scraping service that provides advanced capabilities for extracting content from websites. This module enables loading and processing web content through the FireCrawl API.

This module provides a sophisticated web crawler that can:

• Scrape single web pages
• Crawl entire websites
• Extract structured data
• Handle JavaScript-rendered content
• Process content with text splitters
• Customize metadata extraction
• Support multiple operation modes

Inputs

Required Parameters

• URL: The webpage or website URL to process
• Connect Credential: FireCrawl API credentials
• Mode: Choose between:
  • Scrape: Single page extraction
  • Crawl: Multi-page website crawling
  • Extract: Structured data extraction

Optional Parameters

• Text Splitter: A text splitter to process the extracted content
• Scrape Options:
  • Include Tags: HTML tags to include
  • Exclude Tags: HTML tags to exclude
  • Mobile: Use mobile user agent
  • Skip TLS Verification: Bypass SSL checks
  • Timeout: Request timeout
• Additional Metadata: JSON object with additional metadata
• Omit Metadata Keys: Comma-separated list of metadata keys to omit

Outputs

• Document: Array of document objects containing metadata and pageContent
• Text: Concatenated string from pageContent of documents

Features

• Multiple operation modes
• Advanced scraping options
• Structured data extraction
• JavaScript rendering
• Mobile device emulation
• Custom timeout settings
• Error handling

Operation Modes

Scrape Mode

• Single page processing
• Main content extraction
• Format selection
• Custom tag filtering

Crawl Mode

• Multi-page crawling
• Subdomain handling
• Sitemap processing
• Link extraction

Extract Mode

• Structured data extraction
• Schema-based parsing
• LLM-powered extraction
• Custom extraction prompts

Document Structure

Each document contains:

• pageContent: Extracted content in markdown format
• metadata:
  • title: Page title
  • description: Meta description
  • language: Content language
  • sourceURL: Original URL
  • Additional custom metadata

Notes

• Requires valid FireCrawl API key
• Supports multiple content formats
• Handles rate limiting
• Job status monitoring
• Error handling and retries
• Customizable request options
• Memory-efficient processing