Querylist Laravel Package

Getting Started

QueryList is a standalone PHP package — there is no service provider or config file to publish. Install it directly:

composer require jaeger/querylist

The best entry point is the fluent QueryList::get() API, which mirrors the flow of a scraper you'd write by hand: fetch → select → extract → iterate.

use QL\QueryList;

$data = QueryList::get('https://news.ycombinator.com')
    ->rules([
        'title' => ['a.titlelink', 'text'],
        'link'  => ['a.titlelink', 'href'],
    ])
    ->range('.itemlist .athing')
    ->query()
    ->getData();

// $data is an Illuminate\Support\Collection

For a Laravel developer, the fastest first use case is a one-off Artisan command that pulls a page and maps rows into an Eloquent model. Reach for QueryList::get() for HTTP, ->rules() to declare extraction, ->range() to scope rows, and ->query()->getData() to get a Collection you can ->map() and ->each() into the DB. Everything downstream of getData() is normal Laravel.

If you only have HTML (e.g., from Http::get()->body() or a stored file), use QueryList::html($html) instead — the extraction API is identical.

Implementation Patterns

Rule-driven extraction. Define a rules map once and reuse it as the canonical schema for a page type. Rules take the form [selector, attribute] where the attribute can be text, html, attr('href'), or omitted for outer HTML. Keeping rules in a dedicated class or config makes selectors testable independently of fetching.

Chunked crawls with concurrency. For lists of URLs, ->concurrency() (curl_multi under the hood) is the main scaling primitive. Fan out with a callback that receives each QueryList instance, then persist per callback:

QueryList::get('https://example.com/list')
    ->rules(['detail' => ['a.item', 'href']])
    ->query()
    ->getData()
    ->pipe(function ($urls) {
        return QueryList::getInstance()
            ->concurrency($urls->pluck('detail')->all())
            ->success(fn (QueryList $ql) => $ql->rules([...])->query()->getData())
            ->send()
            ->getData();
    });

For anything larger than a few hundred URLs, move concurrency into a queued Job and let Laravel handle throttling and retries; QueryList's concurrency is in-process and blocks the worker.

Pagination via page(). The ->page() helper handles the common "next page" pattern by passing an array of page descriptors and a callback that extracts the next URL. This is useful for index crawls that fit within a single job's time budget; beyond that, persist the "next URL" as a cursor and re-dispatch a job.

Pipeline integration. Treat QueryList as the fetch+parse layer, then hand off:

• ->getData()->each(fn ($row) => MyModel::updateOrCreate(['url' => $row['url']], $row))
• ->getData()->map(fn ($row) => new ArticleDto($row))->toArray() before queueing
• Normalize URLs with url()->to() or a small helper, since QueryList returns relative href values verbatim.

Encoding conversion. Set ->encoding('UTF-8', 'GBK') (target, source) before calling ->query(). This is essential for Chinese and other legacy-encoded sites and must happen before extraction.

Plugins as cross-cutting concerns. Implement QueryList\Plugin\QueryListPlugin and register with ->use(MyPlugin::class) for things like proxy rotation, request logging, or header injection. Plugins can hook the fetch and query lifecycle, which is the idiomatic extension point — prefer this over subclassing.

Laravel-friendly wrappers. Wrap repeated scrapes in a small Scraper service that takes a URL, applies a shared QueryList instance configured with user agent, timeout, and default headers, and returns a typed DTO Collection. Centralizing the ->userAgent(), ->timeout(), and ->header() calls avoids drift between jobs.

Gotchas and Tips

No Laravel integration ships with the package. Scores and dependents are low, so don't expect a service provider, facade, or config. You configure behavior per-instance. Budget time for a thin wrapper class rather than hunting for official Laravel docs.

phpQuery is bundled and dated. The default phpQuery engine has CSS selector limitations and has historically had PHP 8 deprecation noise. If you hit selector gaps or type errors, swap the query engine to Dom (Symfony DomCrawler) or Goutte via ->setQueryEngine(). This is the first debugging lever when a selector "works in the browser but not here."

Always check ->isSuccess() and status codes. Failed requests yield empty data collections, not exceptions, by default. Silent empty crawls are the most common bug; assert on getData()->count() or check $ql->getHtml() length before trusting output.

Encoding ordering matters. Setting encoding after ->query() is too late; set it before rules execute. Also verify your source by inspecting Content-Type headers when data looks garbled — mixing utf-8 with ->encoding('UTF-8','UTF-8') is harmless, but skipping conversion on GBK sites will produce mojibake that only shows up when you inspect the DB.

Relative URLs stay relative. ->attr('href') returns exactly what's in the HTML. Resolve with a helper (Url::to(), parse_url + base URL logic, or league/uri) before persisting or enqueuing.

Memory growth on long runs. Each QueryList instance holds DOM state. For large crawls in a single process, unset($ql) and periodically gc_collect_cycles(). Better: keep jobs short (a few hundred pages) and let the queue recycle the worker.

Respect rate limits yourself. There is no built-in delay, throttling, or robots.txt handling. Add usleep() or a Laravel rate limiter in the job's handle() — this is on you, and it's what gets crawlers blocked.

Config quirks around retries. ->retry(n) exists in some versions but behavior differs; the safest pattern is a Laravel retry() wrapper around the QueryList call, or a queued job with $tries and backoff.

Plugin lifecycle is per-instance. Registering a plugin on a QueryList only affects that instance and instances derived via concurrency/getInstance. If you build a shared helper, apply plugins in one place so all derived instances inherit them.

Version pinning. v3 → v4 broke the API significantly (fluent rules, query()->getData() instead of the old array-returning API). Pin a specific minor version in composer.json and re-run your scrapers in CI — selector regressions in phpQuery or DOMCrawler updates are common.

Extension points to know: QueryListPlugin (cross-cutting), QueryProvider (custom parsing backends, e.g., a headless-browser or JSON-API backend), and the ->query() callback (post-processing). Those three cover nearly every real-world customization without touching internals.