Copycat Laravel Package

Getting Started

Install and confirm it runs on your PHP version first. The package's last release was 2017, so before you build anything on it, do a 10-minute smoke test on your actual PHP runtime (8.2/8.3 are the likely pain points — see Gotchas).

composer require gidlov/copycat

Mental model. Copycat is three things glued together, and knowing which one you're touching saves most debugging time:

1. A fetcher (load() / setUrl() + cURL flag) — swappable, and usually the part you don't want in a Laravel app.
2. A DOMDocument exposed as $cc->dom — a plain PHP object you can reach into at any time.
3. A CSS→XPath translator exposed as $cc->query(), returning a node wrapper with magic getters.

Where to look first. The README is short and the surface area is tiny — read src/Copycat.php and src/Node.php end to end (a few hundred lines). Everything you'll ever need is in those two files; there is no config file, no service provider, no facades.

First use case — one page, a handful of fields:

use Gidlov\Copycat\Copycat;

$cc = new Copycat();
$cc->curl = true;                          // avoid allow_url_fopen
$cc->load('https://news.ycombinator.com/'); // some versions: setUrl() then load()

foreach ($cc->query('a.storylink') as $a) {
    echo $a->text . ' -> ' . $a->href . PHP_EOL;
}

// Magic getters return arrays when the selector matches many nodes
$titles = $cc->query('a.storylink')->text;
$hrefs  = $cc->query('a.storylink')->href;
$src    = $cc->query('img.logo')->attr('src');
$html   = $cc->query('div#content')->html;   // inner/outer HTML of the node

That's the whole day-to-day API: query(), iteration, text/html/attribute access, each() for callbacks. If your first scraper works in 20 lines, you've learned the library.

Implementation Patterns

Split fetching from parsing — this is the single most important pattern. Don't let Copycat make HTTP calls in production code. Fetch with Laravel's HTTP client (retries, timeouts, logging, fakes in tests), then hand the HTML string to Copycat:

class HackerNewsScraper
{
    public function parse(string $html): array
    {
        $cc = new Copycat();
        $cc->dom->loadHTML($html);   // parse-only; no network
        // ... selectors ...
    }
}

// caller
$html = Http::retry(3, 500)->timeout(15)
    ->withHeaders(['User-Agent' => config('scraper.ua')])
    ->get($url)->body();

$rows = app(HackerNewsScraper::class)->parse($html);

Benefits: unit tests need no network, Http::fake() gives you deterministic fixtures, and you can replay saved HTML when a selector breaks.

One class per site, declarative field map. Scrapers rot when selectors live inline in controllers. Keep a map so a selector fix is a one-line diff and non-authors can patch it:

private array $fields = [
    'title' => ['sel' => 'h1.product-title', 'get' => 'text'],
    'price' => ['sel' => 'span.price',       'get' => 'text'],
    'sku'   => ['sel' => 'meta[name=sku]',   'get' => 'content'],
    'image' => ['sel' => 'img.main',         'get' => 'src'],
];

Row-level iteration for listings. When one page yields many records, iterate the container node and run relative sub-queries on each node rather than building giant global XPath expressions:

$rows = [];
foreach ($cc->query('li.result') as $node) {
    $rows[] = [
        'title' => trim($node->query('h2')->text ?? ''),
        'url'   => $node->query('a')->href ?? null,
    ];
}

Fallback chains. Marketing sites ship A/B variants and redesigns. Try selector A, then B, then log:

$price = $cc->query('.price-now')->text
      ?: $cc->query('[data-price]')->attr('content')
      ?: $this->logSelectorMiss('price', $url);

Queue it, throttle it. Copycat has no concurrency control, no delay, no robots.txt awareness. Wrap it in a job (ShouldQueue) with RateLimited middleware or a Redis::throttle lock, and set a per-host delay. Crawls belong on a queue with Bus::batch() for progress, never in a request/response cycle.

Cache and upsert. Cache::remember("scrape:$url", now()->addHours(6), ...) cuts fetch volume enormously for detail pages. For persistence, key on a stable natural key (SKU, canonical URL) and Model::upsert() — scraped data is re-fetched, not appended.

Store raw HTML on every parse. Write to storage/app/scrapes/{sha1(url)}.html (or a scrapes table) for a rolling window. When a selector silently returns nothing three weeks later, having the source you parsed is the difference between a 5-minute and a 5-hour fix.

Reach into $cc->dom / $cc->xpath when CSS isn't enough. Namespace-heavy XML feeds (RSS, sitemaps) and starts-with()/contains() predicates are easier in raw XPath than in the translator's subset. You're not "going around" the library — that's an intended extension point.

Gotchas and Tips

• UTF-8 will be mangled by default. DOMDocument::loadHTML assumes ISO-8859-1 unless the markup declares a charset. Either prepend <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> to the string, or run mb_convert_encoding($html, 'HTML-ENTITIES', 'UTF-8'). Accented text, currency symbols, and CJK all break without this — it's the #1 first-hour bug.
• Suppress libxml noise. Malformed real-world HTML triggers warnings into your logs. Set libxml_use_internal_errors(true) before loading and libxml_clear_errors() after; inspect libxml_get_errors() only when debugging.
• Failures are silent. A non-matching selector gives you an empty node wrapper; ->text yields null/empty string, not an exception. Assert expected counts (abort_if($cc->query('.product')->count() === 0, ...) or throw a SelectorMissException) or your pipeline will happily store nulls forever.
• Selectors are a translated subset, not full CSS. Complex pseudo-classes (:nth-child, :not(), :has()) may be unsupported or behave differently than in a browser. If a selector works in DevTools but not here, drop to $cc->xpath->query('//div[@data-x]') rather than fighting the translator.
• PHP 8.x compatibility is unverified. The 2017 release predates typed properties and the 8.2 dynamic-property deprecation; public flags like $cc->curl may emit deprecation notices depending on how they're declared. Pin the version, run your test suite on the target runtime, and be prepared to fork ("repositories": [{"type": "vcs", ...}]) or migrate to symfony/dom-crawler, Roach, or Browsershot if you hit hard blockers.
• Relative URLs are not resolved. ->href gives you exactly what's in the attribute. Run results through url()->setScheme(...) / League\Uri or parse_url against the base URL before storing.
• No cookies, sessions, proxies, or JS. Logged-in scraping needs a Guzzle cookie jar and your own fetch layer. JS-rendered pages need a headless browser entirely — Copycat will just see an empty shell. Recognize that boundary early; it decides the architecture.
• Politeness is your job. Set a contactable User-Agent, honor robots.txt, respect Retry-After on 429/503, and keep per-host concurrency at 1–2. Getting your app's IP blocked is a library-agnostic problem the library won't warn you about.
• Debug by dumping the DOM, not by guessing. $cc->dom->saveHTML() (or saveHTML($node)) around the region in question, plus $cc->xpath->query('//your/path')->length, resolves almost every "why is this empty" in one pass.
• LGPL-3.0, not MIT. For most applications (unmodified dependency, dynamically used) this is fine, but it's worth a legal check if you ship closed-source and vendor the package. Keep any modifications in separate files or a fork you're willing to publish.
• Design for abandonment. Given the release age, isolate Copycat behind your own thin interface (Scraper::parse(string $html): array). When the day comes to swap the engine, you change one adapter class instead of 40 selectors scattered across the codebase.