vdb/php-spider
A configurable PHP web spider/crawler with a modular architecture for scraping and traversing sites. Define start URLs, URI filters, processors, and post-processors to control crawl depth, skip duplicate pages, and extract or analyze content with custom callbacks.
The package is a plain Composer library — no Laravel service provider, facade, config file, or migrations. Budget an hour to read the README and the src/ tree, then wire it into an Artisan command. It is a lightweight crawler, not a crawling platform: no JavaScript rendering, no built-in robots.txt handling, no distributed queue.
Install:
composer require vdb/php-spider
Check composer.json for PHP and extension constraints. ext-dom is required through symfony/dom-crawler; ext-curl only matters if you use the cURL downloader. Laravel already ships Monolog, so the package's logger hooks work without extra dependencies.
Where to look first:
README.md — canonical examples and constructor signatures.src/Spider.php — the crawl loop and getDownloader(), getDiscoverer(), getQueue().src/Event/SpiderEvents.php — event names for hooks.src/Filter/Uri/, src/PostProcessor/, src/PersistenceHandler/ — the extension points you will actually use.tests/ — fastest way to confirm current constructor arguments and event accessors.First use case: mirror a docs site to disk, then inspect the output before adding filters or database writes.
// app/Console/Commands/CrawlDocs.php
namespace App\Console\Commands;
use Illuminate\Console\Command;
use VDB\Spider\Spider;
use VDB\Spider\PersistenceHandler\FileSystem;
use VDB\Spider\PersistenceHandler\FileSystem\Naming\Sha1;
class CrawlDocs extends Command
{
protected $signature = 'spider:docs {url=https://example.com/docs}';
protected $description = 'Crawl docs pages into storage/app/spider';
public function handle(): int
{
$dir = storage_path('app/spider');
@mkdir($dir, 0775, true);
$spider = new Spider($this->argument('url'));
$spider->getDiscoverer()->setMaxDepth(3);
$spider->addPersistenceHandler(
new FileSystem('file://'.$dir, new Sha1())
);
$spider->crawl();
return self::SUCCESS;
}
}
Run php artisan spider:docs. That gives you a local HTML mirror you can inspect before introducing filters, post-processors, or Eloquent writes.
Run it in a command or queued job, never a controller. Crawls are synchronous, blocking, and potentially unbounded; a web request will time out. Use an Artisan command for ad hoc runs and a queued job for scheduled crawls. Do not serialize the Spider object into a job — it holds an event dispatcher, downloader, and potentially closures. Queue a small DTO (seed URL, max depth, host allowlist) and construct the spider inside handle().
Two-phase persistence is the most reliable pattern. Use PersistenceHandler\FileSystem to mirror pages, then a separate ingest command or job parses the stored HTML with Symfony DomCrawler or symfony/css-selector. This decouples crawling from parsing, makes re-parsing free, and survives extraction changes.
Stream into the database with an event listener. For smaller crawls, register a listener on the post-request or post-persist event and upsert into Eloquent. Inspect src/Event/ for the exact event class and accessors; the shape is usually:
$spider->getDispatcher()->addListener(
SpiderEvents::SPIDER_CRAWL_POST_REQUEST,
function ($event) {
$resource = $event->getArgument('resource'); // verify accessor in src/Event
Page::updateOrCreate(
['url' => (string) $resource->getUri()],
['title' => $resource->getCrawler()->filter('title')->text('')]
);
}
);
Keep the listener fast and idempotent (updateOrCreate), and catch exceptions so one malformed page does not abort the crawl.
Custom post-processors for extraction. PostProcessor\XPath('//h1') is fine for experiments, but a custom PostProcessorInterface implementation (or a callback post-processor) is easier to test and maps cleanly to a DTO. The Resource wraps a DomCrawler, so filter(), filterXPath(), text(), and attr() are available without adding another parsing library.
Custom downloader for auth, headers, retries, and throttling. If the target needs cookies, bearer tokens, proxies, or Laravel HTTP client semantics, implement the package's downloader interface and delegate to Http::withToken(...)->get(...). This is also the right seam for global rate limiting (usleep, or a Redis token bucket), because the package does not throttle by default. Verify the expected response type (PSR-7 vs Symfony response) in src/Downloader/ before wiring it up.
Constrain aggressively. Day-to-day crawls get out of hand fast. Use Filter\Uri\Contains, Filter\Uri\Host or Filter\Uri\Regex, setMaxDepth(), and query-string normalization before anything else. For large sites, replace the default in-memory queue with a QueueManagerInterface backed by Redis or a crawl_urls table so the crawl is resumable and memory-bounded.
Laravel integration shape. Put seed URLs, allowed hosts, depth, and storage paths in config/spider.php. Bind a SpiderFactory in a service provider so commands and jobs share construction. Log through a dedicated Monolog channel (Log::channel('spider')) and schedule with $schedule->command('spider:run')->hourly()->withoutOverlapping(). Use a cache lock as well — two crawls of the same site will fight over the same output directory and database rows.
Good day-to-day uses: internal link audits (crawl your own site, diff against route:list), docs-to-search-index pipelines, legacy content migration, sitemap or link validation, and scheduled monitoring of a small set of pages. Not a fit: JavaScript-rendered SPAs (no headless browser), large-scale distributed crawling, or anything that needs built-in robots.txt compliance and rate limiting.
memory_limit. Set a CLI memory limit in the command and cap depth or page count early; swap the queue manager for large jobs.https://x.com/a, https://x.com/a/, https://x.com/a#frag, and ?utm_source=... can all be treated as distinct. Normalize scheme and host casing, strip fragments and tracking params, and decide on trailing slashes. Otherwise you will re-crawl duplicates and inflate counts.<base href>. The discoverer resolves relative URLs against the current page, but protocol-relative links (//cdn...) and <base> tags are common sources of wrong-host crawls. Add a host filter as a safety net.Content-Type and add a max body size in the downloader. Also set connect and read timeouts — the stream downloader's defaults may be patient.Resource; check $resource->getResponse()->getStatusCode() in listeners before parsing. Parse failures in a post-processor should be caught and logged per URL, not allowed to kill the run.//div queries. Prefer CSS selectors (filter()) when possible and test against saved fixtures. Watch encodings — force UTF-8 if the source is legacy.src/Event/SpiderEvents.php and the event class rather than copying snippets from old blog posts. The dispatcher is Symfony's, so stopPropagation() works if you need to abort a page.FileSystem\Naming\Sha1 avoids path traversal and long filenames but produces opaque files; a URL-based naming strategy is friendlier for debugging. Whichever you choose, keep the URL-to-file mapping in the database if you plan to re-ingest.setMaxDepth, filters, and downloader overrides before crawl(), and do not assume the defaults you saw in the README are the defaults in your installed version — check the constructor.FilterInterface, PostProcessorInterface, DownloaderInterface, QueueManagerInterface, DiscovererInterface, plus Symfony event listeners. Most "the package cannot do X" problems are solved at one of these seams rather than by forking.How can I help you explore Laravel packages today?