Weave Code
Laravel Laravel Package
roach-php/laravel
Laravel integration for Roach PHP, a Scrapy-inspired web scraping framework. Run spiders as Artisan commands, configure crawlers and pipelines, and use Laravel's container, config, queues, and scheduling to build maintainable scrapers inside your app.
Laravel adapter for Roach, the complete web scraping toolkit for PHP.
Frequently asked questions about Laravel
- How do I install roach-php/laravel in a Laravel app?
- Install it with composer require roach-php/laravel, then publish the package config using php artisan vendor:publish so a roach.php config file lands in your config directory. There are no database migrations, routes, or middleware to register, so integration is limited to Composer and config. After that you can scaffold your first spider with the provided Artisan command and start defining start URLs and parse callbacks.
- Which Laravel and PHP versions does roach-php/laravel support?
- The package targets modern Laravel applications, with Laravel 10, 11, and 12 supported on PHP 8.2 and above. The 3.2.0 release from March 2025 specifically added Laravel 12 and PHP 8.4 support, making it viable for teams on current stacks. If you are on an older Laravel or PHP version, check the composer.json constraints before installing, and run your own regression tests after upgrading.
- How do I create and run a spider with this Laravel package?
- You scaffold a Spider class using the package's Artisan command, then define entry URLs plus parse and parseItem callbacks that describe how to extract and transform data. Spiders can be executed directly from an Artisan command during development or dispatched from a queued job in production. Because the spider is just a class resolved from Laravel's container, you can inject services, read config, and log through your normal application channels.
- Does roach-php/laravel support queues and the Laravel scheduler?
- Yes. The recommended production pattern is to dispatch a crawl from a queued job so scraping work runs on your existing queue workers instead of blocking a request or command. You can then register that job on Laravel's scheduler for recurring crawls, and use the package config to control concurrency and politeness settings. This lets you reuse your existing queue infrastructure, retries, and failed-job handling for scraping workloads.
- Can roach-php/laravel scrape JavaScript-heavy sites and SPAs?
- No. Roach is an HTTP-client-level scraping toolkit, so it fetches and parses the HTML returned by the server and does not execute JavaScript. Targets built as single-page apps, or pages that lazy-load content behind client-side rendering, will return empty or partial markup. For those sites you need a separate headless-browser layer such as Chrome or Playwright alongside Roach, or you should pick a different tool entirely.
- How does Roach handle proxies, CAPTCHAs, and anti-bot protection?
- It does not solve these out of the box. Cloudflare, DataDome, Akamai, 403 and 429 responses, and CAPTCHA challenges are the leading causes of production scrape failures, and Roach alone will not get past them. You can add custom downloader middleware for headers, retries, and rate limiting, and you can route traffic through a proxy tier, but CAPTCHA handling and bot mitigation require separate infrastructure and, often, a legal review of whether bypassing them is acceptable.
- How do I test a Roach spider in a Laravel test suite?
- Keep parsing logic in small, pure methods that accept an HTML string or a response object so you can feed them fixture markup and assert on the extracted items without any network calls. Wrap the crawler execution behind a service or job you can fake in tests, and use Laravel's HTTP faking utilities for any surrounding HTTP interactions. Testing selectors against saved fixtures also gives you fast feedback when a target site changes its markup.
- Is roach-php/laravel production-ready and actively maintained?
- It is usable in production, but treat it as a moderate-risk dependency rather than a core framework package. It has a modest star count and a small contributor base, so a maintainer step-away would leave you maintaining a fork. Pin the version in composer.lock, keep an internal owner for the dependency, and have a fork or replacement strategy documented before you build critical pipelines on it.
- How does roach-php/laravel compare to Scrapy or Guzzle with DomCrawler?
- Scrapy is a full Python scraping framework and lives outside your Laravel app, which means a separate runtime, deployment, and data handoff. Guzzle plus DomCrawler is quick to start but usually decays into ad-hoc command scripts with no reusable lifecycle or pipeline abstraction. Roach is the middle ground: Scrapy-inspired spiders, item pipelines, and middleware that plug directly into Laravel's container, config, queues, and scheduler.
- How do I store scraped items in my database with Roach?
- Define an item pipeline that receives each parsed item and persists it through your normal Laravel channels, such as an Eloquent model, a repository, or a dispatched job. Keeping persistence in a pipeline rather than inside parse callbacks makes the spider reusable and keeps writes testable and transactional. For high-volume crawls, consider queuing the persistence step so slow database writes do not stall the crawl itself.
Weaver
How can I help you explore Laravel packages today?