Firecrawl Sdk Laravel Package

Getting Started

1. Install and key up

composer require firecrawl/firecrawl-sdk

FIRECRAWL_API_KEY=fc-xxxxxxxxxxxxxxxx

// config/services.php
'firecrawl' => [
    'key' => env('FIRECRAWL_API_KEY'),
    'base_url' => env('FIRECRAWL_BASE_URL', 'https://api.firecrawl.dev'),
],

2. Where to look first

Open src/ before README.md. With 2 stars and 0 dependents this is a thin, early-stage wrapper — the constructor signature, method names, and response DTOs are the real contract, and the README may lag behind. Look for:

• the main client class (constructor takes API key + optional base URL)
• the scrape / crawl / map / search / extract methods
• how errors surface (exception class vs. ->success flag on the response)

3. First use case: URL → Markdown

The 80% case is turning a page into clean Markdown for storage, search, or LLM context.

use Firecrawl\FirecrawlClient; // confirm exact class name in src/

$client = app(FirecrawlClient::class);

$result = $client->scrape('https://example.com/blog/post', [
    'formats' => ['markdown'],
    'onlyMainContent' => true,
]);

$markdown = $result['data']['markdown'] ?? $result->markdown;

Get that working in php artisan tinker before building anything around it. Everything else in the SDK is a variation on this call.

Implementation Patterns

Bind the client as a singleton

The SDK client is stateless HTTP, so don't re-instantiate per call. Register it once:

// app/Providers/AppServiceProvider.php
$this->app->singleton(FirecrawlClient::class, fn () => new FirecrawlClient(
    apiKey: config('services.firecrawl.key'),
    baseUrl: config('services.firecrawl.base_url'),
));

Then inject it anywhere. This also gives you a single seam to mock in tests.

Wrap it in your own service class

Given the package's maturity (v0.x-style churn is likely), don't let FirecrawlClient leak into controllers. Put a WebScraper / SiteCrawler service in front of it so an SDK breakage is a one-file fix:

class PageScraper
{
    public function __construct(private FirecrawlClient $client) {}

    public function toMarkdown(string $url): string
    {
        $res = retry(3, fn () => $this->client->scrape($url, [
            'formats' => ['markdown'],
            'onlyMainContent' => true,
        ]), 1000);

        return $res['data']['markdown'];
    }
}

Always queue scrapes and crawls

Scrapes take 2–30s (JS rendering, PDF parsing), and crawls take minutes. A synchronous call in a controller will blow through max_execution_time on anything real.

class ScrapePageJob implements ShouldQueue
{
    public int $timeout = 120;
    public int $tries = 3;
    public array $backoff = [10, 60, 300];

    public function __construct(public string $url, public int $documentId) {}

    public function handle(PageScraper $scraper): void
    {
        $markdown = $scraper->toMarkdown($this->url);
        Document::whereKey($this->documentId)
            ->update(['markdown' => $markdown, 'scraped_at' => now()]);
    }
}

Use map before crawl

crawl is the expensive, slow, credit-hungry call. Run map first to see the URL inventory, filter it (drop /tag/, /author/, query-string permutations), then crawl a scoped includePaths list. This is the single biggest cost lever.

$urls = $client->map('https://docs.example.com')['links'];
$paths = collect($urls)
    ->filter(fn ($u) => str_contains($u, '/docs/'))
    ->map(fn ($u) => parse_url($u, PHP_URL_PATH))
    ->values()->all();

$job = $client->crawl('https://docs.example.com', [
    'includePaths' => $paths,
    'limit' => 200,
    'scrapeOptions' => ['formats' => ['markdown'], 'onlyMainContent' => true],
]);

Crawl is asynchronous — poll or webhook

crawl returns a job ID immediately. Either poll getCrawlStatus($id) in a scheduled job until status === 'completed', or register a webhook if the SDK version supports it and let Laravel handle it via a route. Polling is simpler but burns worker time; for anything long-running, prefer the webhook endpoint in your app.

Cache aggressively

The same URL does not change often. Cache by URL + options hash for hours or days; it's the difference between a $5/month and a $500/month bill.

$markdown = Cache::remember(
    'fc:'.sha1($url.json_encode($options)),
    now()->addDay(),
    fn () => $this->scraper->toMarkdown($url)
);

Batch for URL lists

If you have a known set of URLs, prefer the batch-scrape endpoint over a foreach of individual scrapes — fewer round trips, better throughput, and progress reporting.

Combine search + scrape for research flows

search gives you URLs and snippets; feed the top N into scrape to get full content. That's a natural fit for a queued ResearchTopicJob that writes results into an articles table.

Structured extraction

For schema-shaped output (pricing tables, product specs, article metadata), use extract with a JSON schema rather than parsing markdown with regex. Keep the schema small and flat — large schemas are where extraction quality degrades and costs climb.

Gotchas and Tips

It's a hosted API, not a library. Every method is a paid network call against Firecrawl's servers. There is no local fallback. Treat it like Stripe: retries, idempotency, timeouts, and cost monitoring around every call.

Pin the version and read the diff on upgrade. At 2 stars and 0 dependents, this package has essentially no community pressure testing it. Use "firecrawl/firecrawl-sdk": "0.0.x" with a caret only if you're comfortable, and check composer.lock diffs. Better: your service wrapper means an upgrade is a one-day task, not a rewrite.

Response shape is not a DTO you control. Older/current versions may return arrays, stdClass, or typed response objects depending on the endpoint. Always null-coalesce ($result['data']['markdown'] ?? '') and never assume a key exists. Log the raw response on unexpected shapes so you can see what the API returned.

Handle rate limits and 429s explicitly. Firecrawl enforces concurrency limits per plan. Use retry() with exponential backoff, and cap concurrent scrape jobs via a rate-limited queue or a WithoutOverlapping middleware on the crawl job — otherwise a bulk import will just 429-loop.

Crawl can silently return partial results. status: completed doesn't mean every URL succeeded. Inspect per-page errors; a JS-heavy page can fail while the crawl as a whole "succeeds." Persist failures and retry them individually.

Markdown is truncated on large pages. onlyMainContent helps by stripping nav/footer, but huge pages still get cut. Check for a truncation flag or length threshold in the response and handle oversize content (chunk it, or re-scrape a specific section).

Respect robots.txt, ToS, and rate etiquette on the target site. Firecrawl crawls politely by default, but you own the legality of what you crawl. Never point a crawl at a site you don't have permission to index, and honor includePaths/excludePaths to avoid hammering login pages, search results, or infinite pagination.

Testing: mock the client, not HTTP. Don't Http::fake() and hope the SDK routes through Laravel's HTTP client — it almost certainly uses Guzzle directly. Mock your own PageScraper/WebScraper service in feature tests, and write one narrow unit test against FirecrawlClient with a Guzzle MockHandler if you need to verify request shaping.

Store the raw response, not just the markdown. Keep the full JSON (or a json column) alongside the extracted markdown. When the SDK changes shape or you realize you need metadata.title or links later, you won't have to re-pay for re-scrapes.

Extension points are thin — plan to fork. There's likely no Laravel service provider, no config file to publish, no events, no middleware. Don't wait for them. Your own singleton binding + service wrapper + events (PageScraped, CrawlCompleted) is the right integration layer, and it's ~50 lines.

Timeouts matter more than you think. Default Guzzle timeouts (~30s) are borderline for scrape and too short for crawl status checks on big sites. Set an explicit longer timeout on the client constructor or per-request options, and make sure your ShouldQueue::$timeout exceeds it.

Watch credits in logs. Log the credit cost or URL count per call. A single accidental full-site crawl without a limit can consume a month of credits in one job — limit and maxDepth are not optional.