You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0811001
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: README.md
+44-24Lines changed: 44 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -210,7 +210,7 @@ When using a custom API URL (anything other than `https://api.firecrawl.dev`), a
210
210
211
211
### `scrape` - Scrape URLs
212
212
213
-
Extract content from any webpage. Pass multiple URLs to scrape them concurrently -- each result is saved to `.firecrawl/` automatically.
213
+
Extract content from any webpage. A single URL writes to stdout unless `-o` is supplied. Pass multiple URLs to scrape them concurrently; without output flags, successful results are saved to `.firecrawl/`, while failed URLs are reported on stderr without per-URL files. With `--json` or `-o <path>`, results form one JSON array in input order, written to stdout or the requested file. Each item contains `url`, `success`, and the full `data` (including metadata) or an `error`. `--pretty` indents the array. Any failed URL makes the command exit nonzero, after successful results and errors have been saved.
Search prints the returned Search ID and credit usage to stderr when available, keeping stdout suitable for piping. `--json` and `--pretty` preserve the response metadata even when no results match; `-o` also saves empty results.
350
+
348
351
#### Search Options
349
352
350
353
| Option | Description |
@@ -364,6 +367,7 @@ firecrawl search "AI data tools"
364
367
|`--only-main-content`| Include only main content when scraping (default: true) |
Search and URL scrape JSON output add a root `receipt` while preserving existing response fields; multi-URL JSON has a `receipt` on each result item. Raw text output stays unchanged. Available receipt fields are:
924
+
925
+
-`creditsUsed`: actual credits reported by the response, including zero; absent means unknown, not free. A tool's catalog price is not a charge receipt.
926
+
-`requestId`: client idempotency ID for Alexandria calls.
927
+
-`operationId` and `operationType` (`search` or `scrape`): the returned server operation ID and its kind, for tracing the operation.
928
+
929
+
These identifiers serve different purposes. `--request-id` controls Alexandria retry identity; it is not supported for ordinary URL scrape. Diagnostics print available IDs, credits, and retry timing to stderr. Keep stderr separate from JSON stdout; `2>&1` combines them and is not parseable JSON.
930
+
931
+
With `--json` or `-o`, failed search/scrape calls write structured error output before exiting nonzero. A single scrape failure writes JSON even when the requested filename ends in `.md`. Multi-URL explicit output preserves successful results and failures in input order. Inspect the exit code and error fields before treating a saved file as usable content. Missing receipt fields do not establish whether a timed-out operation was billed.
932
+
933
+
On a rate limit, wait at least the returned retry delay when available; otherwise use bounded exponential backoff. API keys on the same team share limits, which vary by plan and endpoint. See the Alexandria retry guidance below for provider executions.
934
+
935
+
For URL scrape, `--timeout <milliseconds>` sets the server-side scrape timeout (the SDK allows transport overhead) and `--max-age 0` requests fresh content. Freshness does not guarantee the source returns a successful page, and a timeout is not proof that upstream work stopped. Inspect returned status/error metadata as well as content.
936
+
917
937
### Format Behavior
918
938
919
939
-**Single format**: Outputs raw content (markdown text, HTML, etc.)
The cap applies to each PDF, not the whole command or total credits. Extra formats and options can add charges. The CLI does not quote page counts or costs before execution. Use JSON output to inspect the returned `metadata.numPages` (parsed), `metadata.totalPages` (document total), and `metadata.creditsUsed` when present; a smaller parsed count means the result is partial.
49
49
50
+
## Receipts and recovery
51
+
52
+
Use `--json` to preserve metadata and the additive `receipt` (at the root for a single scrape, on each result item for multiple URLs). `receipt.creditsUsed` is actual returned usage, including zero; missing means unknown. Existing `metadata.creditsUsed` remains available when returned. `receipt.operationId` identifies the server scrape; Alexandria `receipt.requestId` is a separate client idempotency ID. Available IDs, credit usage, and retry timing print to stderr. Keep stderr separate from JSON stdout.
53
+
54
+
Failures with `--json` or `-o` write structured errors before exiting nonzero, even if the filename ends in `.md`. Inspect the exit code and saved error/status fields before using the content. A successful transport response can still contain a refused or unsuccessful page; do not treat it as task completion or infer a refund.
55
+
56
+
- Use `--timeout <milliseconds>` to set the server-side scrape timeout; the SDK allows transport overhead. A timeout does not prove the operation stopped or cost zero credits.
57
+
- Use `--max-age 0` when fresh URL content is required. This does not guarantee the source page succeeds.
58
+
- On rate limits, honor the returned retry delay when available, otherwise use bounded exponential backoff. Limits are shared across a team's keys and depend on plan and endpoint.
59
+
- For Alexandria, keep the same `--request-id` while an operation is unresolved. A completed failure can replay under the same ID; starting a new attempt requires a new ID and may charge again. Never rotate IDs automatically. Ordinary URL scrape does not support `--request-id`.
60
+
50
61
## Tips
51
62
52
63
-**Prefer plain scrape over `--query`.** Scrape to a file, then use `grep`, `head`, or read the markdown directly — you can search and reason over the full content yourself. Use `--query` only when you want a single targeted answer without saving the page (costs 5 extra credits).
53
64
-**Scrape handles static pages and JS-rendered SPAs.** Escalate to `interact` when the page needs interaction (clicks, form fills, pagination) or scrape misses content.
54
-
- Multiple URLs are scraped concurrently — check `firecrawl --status` for your concurrency limit. This mode saves markdown only and ignores `-o`; other requested formats are dropped. If markdown wasn't requested, the whole JSON response is written into the `.md` file.
65
+
- Multiple URLs are scraped concurrently. Use `--json` for an ordered JSON array on stdout or `-o results.json` to save it. Each item contains `url`, `success`, and full `data` with metadata or an `error`; any failed URL makes the command exit nonzero. Without either flag, successful results are saved under `.firecrawl/` as markdown when available, otherwise JSON in a `.md` file. Failed URLs are reported on stderr without creating per-URL files. Check `firecrawl --status` for your concurrency limit.
55
66
- Single format outputs raw content. Multiple formats (e.g., `--format markdown,links`) output JSON.
56
67
- Always quote URLs — shell interprets `?` and `&` as special characters.
Copy file name to clipboardExpand all lines: skills/firecrawl-search/SKILL.md
+11-3Lines changed: 11 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -28,7 +28,7 @@ Run `firecrawl search --help` for the full option list.
28
28
29
29
`--categories developer` weighs the developer index beside ordinary web results in this same call (no passage control, no index filters). `--categories research` is a website filter, not the paper index. Dedicated skills: [firecrawl-developer-index](../firecrawl-developer-index/SKILL.md) and [firecrawl-research-index](../firecrawl-research-index/SKILL.md).
30
30
31
-
**Done when:**results are saved under `.firecrawl/`, verified non-empty, processed for the request, and one feedback event is sent within the time window (unless opted out).
31
+
**Done when:**the response is saved under `.firecrawl/`, results or an empty result set have been inspected and handled for the request, and eligible feedback is sent within the time window (unless opted out).
32
32
33
33
## Alexandria in normal search
34
34
@@ -40,6 +40,14 @@ Use `find-tools` only for an explicitly requested tool set or a missing contract
40
40
41
41
If no returned tool covers the country/market/segment or required inputs, continue with ordinary web results. Do not exhaust the catalogue or pay for adjacent tools just to probe coverage. `--sources web` explicitly opts out of Alexandria; `--sources web --domain-tools` retains domain matches only.
42
42
43
+
## Receipts and failures
44
+
45
+
JSON output preserves the response and additive `receipt`, including empty results. Read `receipt.creditsUsed` for actual reported usage (zero is valid; missing means unknown) and `receipt.operationId` with `operationType: "search"` for the server search ID. Existing `id` and `creditsUsed` fields remain available. Search IDs identify results; they are not Alexandria client idempotency IDs.
46
+
47
+
Available IDs, credits, and retry timing print to stderr. Keep stderr separate from JSON stdout. Failed calls with `--json` or `-o` write structured errors before exiting nonzero; check the exit code and `success` before using the file. Empty successful searches still write the requested output and retain their metadata.
48
+
49
+
On rate limits, wait at least the returned retry delay when available; otherwise use bounded exponential backoff. API keys on one team share limits, which vary by plan and endpoint. Do not invent a universal requests-per-minute quota or infer zero billing from a missing receipt.
50
+
43
51
## Tips
44
52
45
53
-**`--highlights` on by default:** results are query-relevant excerpts, not full-page snippets. Use `--no-highlights` for the original snippets.
@@ -66,13 +74,13 @@ Search costs 2 credits. After you've actually used the results (or decided they
66
74
-**Idempotent:** re-submitting for the same search id returns success but no extra refund.
67
75
-**`--silent &`** is the right pattern — exit code 0 even on failure, so a rejected/expired call never crashes your pipeline.
68
76
69
-
Verify the search returned results before reading its `id`. Zero-result searches write no output file, so the file may be missing — or left over from an earlier search. The guard below skips feedback when the file is missing or has zero results; call `search-feedback` only inside it:
77
+
Verify the search succeeded before reading its `id`. Empty successful searches preserve their JSON output; failed searches contain error output. The guard below only sends feedback for a successful response with an ID and nonempty results; call `search-feedback` only inside it:
70
78
71
79
```bash
72
80
# Send once per search. Rate honestly and replace the placeholder with the
73
81
# rating that matches what actually happened. The two fields shown
74
82
# satisfy the substantive-content rule for every rating.
75
-
if SEARCH_ID=$(jq -er 'select(any(.data[]; length > 0)) | .id' .firecrawl/search-react-hooks.json);then
83
+
if SEARCH_ID=$(jq -er 'select(.success == true and any(.data[]; length > 0)) | .id' .firecrawl/search-react-hooks.json);then
0 commit comments