Evomi Scraper Laravel Package

Getting Started

Install and configure (5 minutes):

composer require pisotskyi-el/evomi-scraper
php artisan vendor:publish --tag=evomi-scraper-config

Add your credentials to .env:

EVOMI_SCRAPER_API_KEY=your-key
EVOMI_SCRAPER_BASE_URL=https://scrape.evomi.com
EVOMI_SCRAPER_TIMEOUT=60

Where to look first, in this order:

1. config/evomi-scraper.php — the real API surface of the wrapper. Every option you'll fight with (timeout, retries, default country, JS rendering) is decided here.
2. The service provider's bindings — this tells you whether you should inject a class or resolve a facade, and whether you can swap the client in tests.
3. src/DTO/ — the request/response objects. These encode the request contract (target URL, geo, render mode, session) and the response contract (status, body, headers, cost/credits). Read these before writing any code; they're the documentation.
4. The exception classes — knowing what throws vs. what returns a failed DTO saves you an afternoon.

First use case — fetch a page's HTML through the API:

use PisotskyiEl\EvomiScraper\Facades\EvomiScraper;

$response = EvomiScraper::scrape('https://example.com/product/123');

if ($response->successful()) {
    $html = $response->body();
}

The wrapper earns its keep when you move from "did I get HTML" to "did I get HTML from a specific country, with JS rendered, on a sticky session, at a predictable cost." That's the second call, not the first.

Implementation Patterns

Inject, don't facade, in application code. Facades are fine in tinker and one-off commands, but the client should be constructor-injected behind a thin wrapper of your own so you can stub it:

final class ProductPageFetcher
{
    public function __construct(private EvomiScraperClient $client) {}

    public function fetch(string $url): ScraperResponse
    {
        return $this->client->scrape(
            ScraperRequest::make($url)
                ->country('de')
                ->renderJs(true)
                ->session('product-'.$this->shard($url))
        );
    }
}

Treat the request DTO as the configuration object. Build it with named constructors/immutable setters, hold it in a variable, log it before dispatching. When a scrape fails, the request is almost always the interesting thing — geo, render mode, session stickiness — not the response.

Wrap every call in a job. Scraping is slow (2–30s with rendering) and expensive. Controllers should never call this package directly.

final class ScrapeProduct implements ShouldQueue
{
    use InteractsWithQueue, Queueable, Batchable;

    public int $tries = 3;
    public int $backoff = [10, 60, 180];
    public int $timeout = 120;

    public function __construct(public int $productId, public string $url) {}

    public function handle(ProductPageFetcher $fetcher): void
    {
        $response = $fetcher->fetch($this->url);

        if ($response->failed()) {
            throw new ScrapeFailedException($this->url, $response->status());
        }

        ProductSnapshot::create([
            'product_id' => $this->productId,
            'html' => $response->body(),
            'cost' => $response->cost(),
        ]);
    }
}

Batch() these and you get free progress tracking for a crawl; Bus::batch([...])->allowFailures()->dispatch() is the pattern most teams land on within a week.

Rate-limit at the job level, not the HTTP level. Evomi bills per request and enforces concurrency. Use RateLimited middleware or Redis::throttle('evomi')->allow(10)->every(1) in the job's middleware(). Doing this inside the client hides the backpressure from Horizon, which is where you'll actually want to see it.

Persist raw responses, then parse lazily. Store the body (disk or a longText/JSON column) with the request hash. Selectors change constantly; you want to re-parse yesterday's HTML without re-paying for it. This also makes parser bugs debuggable without hitting the API.

Sticky sessions for multi-step flows. Any flow that needs a cart, a paginated listing, or a login must reuse the same session id across requests. Generate one per logical crawl unit and pass it through your DTO — the wrapper will forward it, but it won't invent one for you.

Testing:

Http::fake([
    'scrape.evomi.com/*' => Http::response('<html>ok</html>', 200, ['X-Cost' => '1']),
]);

$this->artisan('scrape:products')->assertSuccessful();

Fake at the HTTP layer rather than mocking the wrapper. The DTO mapping is where bugs live, and mocking the client skips exactly that code. Add one contract test per DTO field you actually use.

Gotchas and Tips

config:cache will bite you first. Publish the config, set values via env() only inside the config file, and re-run php artisan config:cache in deploy. Reading env('EVOMI_SCRAPER_API_KEY') anywhere else returns null in production.

Retries can double-bill. Laravel's ->retry() on a POST is not idempotent from the provider's perspective. If you retry, make sure you're retrying on connection timeouts and 5xx only — a 200-with-empty-body is a product failure (blocked, captcha, wrong country), not a transport failure, and retrying it burns credits for the same result. Inspect the response DTO before deciding.

Timeouts are two different numbers. connect_timeout should be short (5s); the total timeout must exceed your rendering budget (JS-rendered pages routinely take 20–40s). A default 30s timeout produces a flaky 5–10% failure rate that looks like the provider's fault and isn't.

successful() is not the same as "got what I wanted." Anti-bot challenges, soft-blocks, and consent walls come back as HTTP 200 with the wrong HTML. Validate with a cheap structural assertion (a known selector, a JSON key, a minimum body length) before you trust the payload — otherwise you'll silently write captcha pages into your database.

Response shape is conditional. With structured/JSON extraction enabled the body isn't HTML; don't unconditionally run DOMDocument over it. Branch on the content type in the response DTO and fail loudly on an unexpected one.

Memory in queue workers. Holding several megabytes of HTML per job across hundreds of jobs will balloon a worker. Either unset() aggressively, keep bodies on disk with a path reference, or run scrapes in a dedicated queue with --max-jobs=50 so workers recycle.

Sessions are sticky and therefore expensive to leak. Reusing one session across unrelated jobs serializes your throughput on one IP. Namespace session ids by crawl run and let them expire — don't use a single global session because "it worked in testing."

Extension point is the container binding. If you need to add headers, logging, or a fallback provider, bind your own implementation of the package's client contract in a service provider rather than subclassing. That keeps the package upgradable and gives you a single seam to add metrics (requests, failures by status, credits spent) — instrument that seam on day one, because per-request cost is the number that surprises finance.

Log the request DTO, redact the key. A log line with url, country, render, session, and status is enough to debug 90% of failures. Never log the raw config array.