Laravel Scrapingbee Laravel Package

Getting Started

Install and configure

composer require ziming/laravel-scrapingbee
php artisan vendor:publish --tag=scrapingbee-config   # check README/provider if the tag differs

Add the key to .env:

SCRAPINGBEE_API_KEY=your_key_here

Then clear cached config (php artisan config:clear) and confirm the key is read from config/scrapingbee.php. The package is a thin Laravel HTTP-client wrapper around ScrapingBee's REST API, so the single most important reference is the ScrapingBee API docs — every option you pass through the facade ends up as a query parameter on the upstream request.

Where to look first

1. config/scrapingbee.php — API key, base URL, and any default parameters.
2. The facade (Ziming\LaravelScrapingBee\Facades\ScrapingBee) and its underlying client class — method signatures mirror the endpoints (get, post, plus Google/Amazon/Walmart/YouTube helpers and AI querying).
3. Your ScrapingBee dashboard — credits, concurrency limit, and request logs. Budget is the main operational constraint with this package.

First use case: rendered HTML for a JS-heavy page

use Ziming\LaravelScrapingBee\Facades\ScrapingBee;

$response = ScrapingBee::get('https://example.com/product/123', [
    'render_js'     => 'true',
    'premium_proxy' => 'true',
    'country_code'  => 'us',
]);

$html = $response->body();

// Cheap structured extraction instead of parsing HTML yourself
$data = ScrapingBee::get('https://example.com/product/123', [
    'extract_rules' => json_encode([
        'title' => 'h1',
        'price' => '.price',
    ]),
])->json();

A good first production task is: one queued job, one URL, fetch HTML, parse with Symfony DomCrawler or extract_rules, and store the result. Once that works, add caching and rate limiting before scaling to lists of URLs.

Implementation Patterns

Wrap the facade in your own service

Don't call the facade from controllers. Bind a small class behind an interface so you can swap providers, mock it in tests, and centralize defaults:

final class ScrapingBeeClient
{
    public function fetch(string $url, array $options = []): \Illuminate\Http\Client\Response
    {
        return ScrapingBee::get($url, array_merge([
            'render_js'    => 'true',
            'country_code' => config('scraping.country', 'us'),
        ], $options));
    }
}

Queue bulk work and rate-limit it

ScrapingBee enforces concurrency per plan and bills per request, so never loop over hundreds of URLs in a web request:

class ScrapeProduct implements ShouldQueue
{
    use Batchable, InteractsWithQueue, Queueable;

    public int $tries = 3;
    public array $backoff = [10, 60, 300];

    public function middleware(): array
    {
        return [(new RateLimited('scrapingbee'))->dontRelease()];
    }

    public function handle(ScrapingBeeClient $client): void
    {
        $response = $client->fetch($this->url, ['render_js' => 'false']);
        // parse and persist
    }
}

Define the scrapingbee limiter in a service provider based on your plan's concurrency. Use Bus::batch() for large URL sets and record failures per URL.

Cache aggressively

Scraping is expensive and slow; cache by URL plus a hash of the parameters so different render/proxy settings don't collide:

$key = 'sb:'.hash('xxh128', $url.json_encode($params));

$html = Cache::remember($key, now()->addHours(6), fn () => $client->fetch($url, $params)->body());

TTL should reflect data volatility: product prices minutes-to-hours, SERPs hours, static pages days.

Prefer structured output over HTML parsing

• extract_rules — CSS-selector-based JSON extraction; cheapest reliable option for stable pages.
• json_response — ask for JSON where supported.
• js_scenario — click, scroll, fill forms, wait for selectors when the page needs interaction. Pass it as a JSON string.
• screenshot / screenshot_full_page — useful for visual regression or when a client disputes what a page looked like.

Use AI querying for schema discovery, not bulk extraction

AI extraction (ai_query, and the package's AI helper) is excellent for one-off or rapidly changing pages: describe the fields in natural language and get JSON back. It is billed at a credit multiplier, so use it to prototype a schema, then convert the result into extract_rules for production volume.

Use the platform helpers where they fit

The Google/Amazon/Walmart/YouTube helpers encode endpoint-specific parameters so you don't hand-roll them:

$serp = ScrapingBee::google('best laravel scraping packages', [
    'country_code' => 'us',
    'nb_results'   => 20,
]);

Prefer these helpers for SERP rank tracking, product/price monitoring, and transcript/metadata pulls — they save you from reverse-engineering request shapes.

Persist raw responses

Store the raw body (S3/database) alongside parsed output. Credits are non-refundable, so when a parser breaks you want to re-parse locally rather than re-scrape. Include the request parameters and timestamp in the record.

Test with Http::fake()

Never hit the real API in CI:

Http::fake([
    '*scrapingbee.com*' => Http::response('<html><h1>Test</h1></html>', 200),
]);

$client->fetch('https://example.com');

Http::assertSent(fn ($request) => str_contains($request->url(), 'render_js=true'));

Gotchas and Tips

• Credits are the real cost model. A basic request, JS rendering, premium proxy, and AI extraction each multiply cost. Set render_js => 'false' wherever possible; most static pages don't need a browser. Log the per-request cost from response headers or the dashboard and alert on anomalies.
• Booleans must be strings. Pass 'true'/'false', not PHP true/false, for parameters like render_js, premium_proxy, and json_response. Laravel's HTTP client may serialize booleans as 1/0, which ScrapingBee won't interpret as expected.
• country_code generally requires premium_proxy. Setting one without the other produces surprising results or errors.
• Timeouts are easy to hit. wait, wait_for, and heavy JS scenarios can take 60–120s. Set an explicit HTTP timeout, run in a queue, and keep max_execution_time in mind for sync calls.
• Retry selectively. Retry 429 (concurrency) and 5xx with exponential backoff. Don't blindly retry 401 (bad key), 402 (out of credits), or target-side errors like 404 — those are still billed and won't succeed on retry.
• A 200 doesn't always mean success. When the target returns a soft error page or an anti-bot challenge, ScrapingBee may still return content. Validate the presence of expected fields, not just the status code.
• $response->json() can fail. With extract_rules or json_response, the content type is usually JSON, but if it arrives as text/html, fall back to json_decode($response->body(), true) and check the error.
• Don't pre-encode the target URL. Pass the full URL as a parameter value; encoding it yourself produces double-encoded requests.
• js_scenario must be valid JSON. Build it with json_encode() from a PHP array and validate before sending; a malformed scenario silently does nothing or errors upstream.
• Config cache bites after env changes. php artisan config:clear (or config:cache in deploy) whenever SCRAPINGBEE_API_KEY changes; otherwise you'll chase phantom 401s.
• The package is intentionally thin. It does not provide retries, caching, batching, or cost tracking — those are your responsibility. Wrap it behind an interface so you can add them centrally and swap providers later.
• Pin the version and watch compatibility. With a small single-maintainer package and a low star count, pin the constraint in composer.json, check Laravel/PHP version support before upgrading, and keep your wrapper interface stable so a package change is a one-file fix.
• Legal and ToS. ScrapingBee handles proxies and CAPTCHAs, but you still own compliance with target sites' terms, robots.txt, rate expectations, and GDPR/data-residency rules. Keep a per-domain policy in config rather than scattering decisions through jobs.
• Extension points. Publish and edit the config for defaults; override the HTTP client behavior by decorating your own service; use Http::fake() for tests; use Laravel's RateLimited middleware rather than hand-rolled sleeps.