firecrawl
Firecrawl是可规模化搜索、抓取网页并支持交互的API,可将网页内容转为干净的Markdown或结构化数据,适配AI智能体使用,提供开源版本与托管服务。
这个项目值得继续研究吗?
Firecrawl是可规模化搜索、抓取网页并支持交互的API,可将网页内容转为干净的Markdown或结构化数据,适配AI智能体使用,提供开源版本与托管服务。
- 解决什么问题
- 企业搭建AI应用、智能体时,获取公开网页数据常遇到反爬限制、动态JS页面无法读取、内容格式杂乱不适合大模型使用的问题,自行搭建爬取工具维护成本高、稳定性差。
- 适合什么团队
- 适合需要批量获取公开网页数据的AI智能体开发团队,以及需要快速采集行业公开信息、竞品动态的业务运营、数据分析团队使用。
- 使用前注意
- 本项目采用AGPL-3.0许可,若自部署后对外提供服务需开源修改后的代码;自部署需自行维护代理池、反爬适配、动态页面解析等相关运维工作,也可选择官方托管服务。
本页用于缩短初步筛选时间,不构成技术、采购或法律结论。 正式使用前请在真实业务数据上验证,并以官方说明与许可证为准。
从官方资料看清能力、部署与采用边界
以下内容依据项目公开 README 或模型卡翻译整理,代码、命令和产品名保持原样。
Firecrawl 项目导读
一、项目定位
Firecrawl是专为AI场景打造的网页数据获取API,核心定位是为AI智能体、大模型应用提供干净可用的网页上下文数据。项目开源开放,同时提供官方托管的SaaS服务,企业无需从零开发复杂的网页爬取工具,通过标准化API即可完成网页搜索、内容抓取、页面交互等操作,大幅降低网页数据获取的门槛。
二、核心能力
- 高可靠网页数据获取:可覆盖96%的公开网页,包括依赖JS动态渲染的复杂页面,官方托管版本自动处理代理轮换、反爬绕过、访问限速、动态内容渲染等底层问题,无需额外配置。
- 大模型适配输出:爬取结果可直接输出为干净的Markdown、结构化JSON、页面截图等格式,内容无冗余杂质,可直接输入大模型使用,减少不必要的token消耗。
- 页面交互能力:支持对已抓取的页面执行点击、搜索、滚动、填写内容、等待加载等操作,模拟真人网页浏览行为,可完成多步操作后的信息提取。
- 智能自动采集:自带Agent能力,无需提前提供目标URL,仅需描述需求即可自动完成网页搜索、页面跳转、信息提取的全流程,还支持自定义数据结构,直接输出符合要求的结构化数据。
- 多场景适配:除基础的单页抓取外,还支持整站爬取、批量URL异步抓取、站点URL快速发现等功能,可满足不同量级的数据采集需求。官方公布数据显示,其跨百万级页面的P95延迟为3.4秒,可支持实时AI应用、动态智能体的使用需求。
三、典型使用方式
使用官方托管服务的流程非常简单,仅需在firecrawl.dev注册账号获取API密钥即可调用,也可先通过官方提供的playground测试功能效果,常用场景包括:
- 全网信息搜索:输入搜索关键词,即可直接返回搜索结果对应页面的完整格式化内容,无需手动逐个访问页面提取信息。
- 定向页面采集:输入任意目标URL,即可获取该页面的格式化内容,适合批量采集产品信息、行业报告、公开公示等内容。
- 多步交互采集:针对需要操作才能获取的信息,比如电商平台商品搜索结果、需要填写表单才能查看的公开内容,可通过交互指令模拟操作流程,提取目标数据。
- 需求驱动自动采集:无需明确目标URL,仅需描述需求,比如“查找Notion的所有定价方案”“提取Firecrawl的创始团队名单”,系统会自动完成全流程采集,还可指定输出格式,直接返回结构化的结果。
四、接入与部署方案
项目提供两种使用方案,企业可根据自身需求选择:
- 官方托管SaaS:无需任何部署工作,注册获取API密钥即可使用,官方负责所有底层运维工作,包括代理维护、反爬适配、功能迭代等。支持Python、Node.js SDK调用,也可通过cURL、CLI工具直接调用,对接成本极低。仅需一行命令即可对接Claude Code等主流AI智能体,也支持对接各类MCP兼容的客户端。
- 开源自部署:可拉取开源代码自行部署,适合有强数据安全合规要求、需要完全掌控数据链路的企业。需注意自部署需要企业自行维护代理池、反爬策略、动态页面渲染引擎等底层组件,运维成本相对较高。
五、许可证与采用建议
本项目采用AGPL-3.0开源许可:如果企业仅调用官方托管的API,或者自部署后仅内部使用,不受许可的传染限制;如果基于开源版本修改后对外提供商业化服务,需要将修改后的代码开源。 采用建议:如果仅需要快速获取网页数据支撑AI应用、智能体开发,优先选择官方托管服务,可大幅降低运维成本,快速上线业务;如果有数据本地化等强合规要求,可选择开源版本自行部署,部署与使用过程中需遵守AGPL-3.0许可的相关要求。
官方资料与来源
- ai
- ai-agents
- ai-crawler
- ai-scraping
- ai-search
- crawler
- data-extraction
- html-to-markdown
- llm
- markdown
- scraper
- scraping
## Quick Start
Sign up at [firecrawl.dev](https://firecrawl.dev) to get your API key. Try the [playground](https://firecrawl.dev/playground) to test it out.
### Search
Search the web and get full content from results.
```python
from firecrawl import Firecrawl
app = Firecrawl(api_key="fc-YOUR_API_KEY")
search_result = app.search("firecrawl", limit=5)
```
<details>
<summary><b>Node.js / cURL / CLI</b></summary>
**Node.js**
```javascript
import { Firecrawl } from 'firecrawl';
const app = new Firecrawl({apiKey: "fc-YOUR_API_KEY"});
app.search("firecrawl", { limit: 5 })
```
**cURL**
```bash
curl -X POST 'https://api.firecrawl.dev/v2/search' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
"query": "firecrawl",
"limit": 5
}'
```
**CLI**
```bash
firecrawl search "firecrawl" --limit 5
```
</details>
Output:
```json
[
{
"url": "http该片段来自项目 README,仅用于初步判断;实际部署请以官方文档为准。
核对上游原始说明节选
The context API to search, scrape, and interact with the web at scale. 🔥
Why Firecrawl?
- Industry-leading reliability: Covers 96% of the web, including JS-heavy pages — no proxy headaches, just clean data (see benchmarks)
- Blazingly fast: P95 latency of 3.4s across millions of pages, built for real-time agents and dynamic apps
- LLM-ready output: Clean markdown, structured JSON, screenshots, and more — spend fewer tokens, build better AI apps
- We handle the hard stuff: Rotating proxies, orchestration, rate limits, JS-blocked content, and more — zero configuration
- Agent ready: Connect Firecrawl to any AI agent or MCP client with a single command
- Media parsing: Parse and extract content from web-hosted PDFs, DOCX, and more
- Actions: Click, scroll, write, wait, and press before extracting content
- Open source: Developed transparently and collaboratively — join our community
---
Feature Overview
Core Endpoints
| Feature | Description | |---------|-------------| | Search | Search the web and get full page content from results | | Scrape | Convert any URL to markdown, HTML, screenshots, or structured JSON | | Interact | Scrape a page, then interact with it using AI prompts or code |
More
| Feature | Description | |---------|-------------| | Agent | Automated data gathering, just describe what you need | | Crawl | Scrape all URLs of a website with a single request | | Map | Discover all URLs on a website instantly | | Batch Scrape | Scrape thousands of URLs asynchronously |
---
Quick Start
Sign up at firecrawl.dev to get your API key. Try the playground to test it out.
Search
Search the web and get full content from results.
from firecrawl import Firecrawl
app = Firecrawl(api_key="fc-YOUR_API_KEY")
search_result = app.search("firecrawl", limit=5)Node.js / cURL / CLI
Node.js
import { Firecrawl } from 'firecrawl';
const app = new Firecrawl({apiKey: "fc-YOUR_API_KEY"});
app.search("firecrawl", { limit: 5 })cURL
curl -X POST 'https://api.firecrawl.dev/v2/search' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
"query": "firecrawl",
"limit": 5
}'CLI
firecrawl search "firecrawl" --limit 5Output:
[
{
"url": "https://firecrawl.dev",
"title": "Firecrawl",
"markdown": "Turn websites into..."
},
{
"url": "https://docs.firecrawl.dev",
"title": "Firecrawl Docs",
"markdown": "# Getting Started..."
}
]Scrape
Get LLM-ready data from any website — markdown, JSON, screenshots, and more.
from firecrawl import Firecrawl
app = Firecrawl(api_key="fc-YOUR_API_KEY")
result = app.scrape('firecrawl.dev')Node.js / cURL / CLI
Node.js
import { Firecrawl } from 'firecrawl';
const app = new Firecrawl({ apiKey: "fc-YOUR_API_KEY" });
app.scrape('firecrawl.dev')cURL
curl -X POST 'https://api.firecrawl.dev/v2/scrape' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
"url": "firecrawl.dev"
}'CLI
firecrawl scrape https://firecrawl.dev
firecrawl https://firecrawl.dev --only-main-contentOutput:
# Firecrawl
Firecrawl helps AI agents search, scrape, and interact with the web.
## Features
- Search: Find information across the web
- Scrape: Clean data from any page
- Interact: Click, navigate, and operate pages
- Agent: Autonomous data gatheringInteract
Scrape a page, then interact with it using AI prompts or code.
from firecrawl import Firecrawl
app = Firecrawl(api_key="fc-YOUR_API_KEY")
result = app.scrape("https://amazon.com")
scrape_id = result.metadata.scrape_id
app.interact(scrape_id, prompt="Search for 'mechanical keyboard'")
app.interact(scrape_id, prompt="Click the first result")Node.js / cURL / CLI
Node.js
import { Firecrawl } from 'firecrawl';
const app = new Firecrawl({apiKey: "fc-YOUR_API_KEY"});
const result = await app.scrape("https://amazon.com");
await app.interact(result.metadata.scrapeId, {
prompt: "Search for 'mechanical keyboard'"
});
await app.interact(result.metadata.scrapeId, {
prompt: "Click the first result"
});cURL
# 1. Scrape the page
curl -X POST 'https://api.firecrawl.dev/v2/scrape' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"url": "https://amazon.com"}'
# 2. Interact with the page (use scrapeId from step 1)
curl -X POST 'https://api.firecrawl.dev/v2/scrape/SCRAPE_ID/interact' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"prompt": "Search for mechanical keyboard"}'CLI
firecrawl scrape https://amazon.com
firecrawl interact exec --prompt "Search for 'mechanical keyboard'"
firecrawl interact exec --prompt "Click the first result"Output:
{
"success": true,
"output": "Keyboard available at $100",
"liveViewUrl": "https://liveview.firecrawl.dev/..."
}---
Power Your Agent
Connect Firecrawl to any AI agent or MCP client in minutes.
Skill
Give your agent easy access to real-time web data with one command.
npx -y firecrawl-cli@latest init --all --browserRestart your agent after installing. Works with Claude Code, Antigravity, OpenCode, and more.
MCP
Connect any MCP-compatible client to the web in seconds.
{
"mcpServers": {
"firecrawl-mcp": {
"command": "npx",
"args": ["-y", "firecrawl-mcp"],
"env": {
"FIRECRAWL_API_KEY": "fc-YOUR_API_KEY"
}
}
}
}Agent Onboarding
Are you an AI agent? Fetch this skill to sign up your user, get an API key, and start building with Firecrawl.
curl -s https://firecrawl.dev/agent-onboarding/SKILL.mdSee the Skill + CLI documentation for all available commands. For MCP, see firecrawl-mcp-server.
---
More Endpoints
Agent
The easiest way to get data from the web. Describe what you need, and our AI agent searches, navigates, and retrieves it. No URLs required.
Agent is the evolution of our /extract endpoint: faster, more reliable, and doesn't require you to know the URLs upfront.
curl -X POST 'https://api.firecrawl.dev/v2/agent' \
-H 'Authorization: Bearer fc-YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
"prompt": "Find the pricing plans for Notion"
}'Response:
{
"success": true,
"data": {
"result": "Notion offers the following pricing plans:\n\n1. Free - $0/month...\n2. Plus - $10/seat/month...\n3. Business - $18/seat/month...",
"sources": ["https://www.notion.so/pricing"]
}
}Agent with Structured Output
Use a schema to get structured data:
from firecrawl import Firecrawl
from pydantic import BaseModel, Field
from typing import List, Optional
app = Firecrawl(api_key="fc-YOUR_API_KEY")
class Founder(BaseModel):
name: str = Field(description="Full name of the founder")
role: Optional[str] = Field(None, description="Role or position")
class FoundersSchema(BaseModel):
founders: List[Founder] = Field(description="List of founders")
result = app.agent(
prompt="Find the founders of Firecrawl",
schema=FoundersSchema
)
print(result.data){
"founders": [
{"name": "Eric Ciarla", "role": "Co-founder"},
{"name": "Nicolas Camara", "role": "Co-founder"},
{"name": "Caleb Peffer", "role": "Co-founder"}
]
}Agent with URLs (Optional)
Focus the agent on specific pages:
result = app.agent(
urls=["https://docs.firecrawl.dev", "https://firecrawl.dev/pricing"],
prompt="Compare the features and pricing information"
)Effort Selection
Set how much reasoning the agent spends on the task:
| Effort | Best For | |--------|----------| | low | Simple lookups on one site | | medium | Multi-step tasks on a few pages | | high | Deep resea
上游文档较长,此处为节选。完整内容见官方项目。