Weave Code
Escargot Laravel Package
terminal42/escargot
A Symfony HttpClient-based crawler framework for PHP that can crawl any HTTP resource, not just websites. Jobs are tracked by ID and backed by a queue, so long crawls can pause and resume.
A web crawler or spider library based on Symfony components
Frequently asked questions about Escargot
- Is terminal42/escargot compatible with Laravel, and which Laravel versions does it support?
- Yes, Escargot is built on Symfony components that Laravel already ships, including HttpClient, Console, and Process, so it integrates cleanly into Laravel applications. It does not declare Laravel-specific version constraints; compatibility follows the Symfony HttpClient and related component versions available in your Laravel release. Check the package composer constraints against your Laravel version before upgrading.
- How do I install terminal42/escargot in a Laravel project?
- Install it with composer require terminal42/escargot. Then register Escargot, your BaseUriCollection, and a queue implementation in a service provider or resolve them inside an Artisan command. For persistent crawls you will usually add a Laravel-native queue adapter or use DoctrineQueue with a compatible database table.
- Can I use Laravel queues or Horizon with Escargot?
- Escargot has its own queue abstraction and tracks every crawl by job ID, so it does not use Laravel queues directly out of the box. You can replace InMemoryQueue or DoctrineQueue with a Laravel-native adapter, but under Escargot 2.0 that adapter must persist and restore redirect destinations via redirected_to. Horizon can run the workers that trigger crawl steps, but the Escargot queue state still needs durable storage.
- What are the breaking changes in Escargot 2.0?
- ExceptionSubscriberInterface::onTransportException() now receives ResponseInterface|null, so existing implementations must update their method signature or fail at class load. DoctrineQueue also requires a nullable redirected_to column in its schema. Redirect semantics changed so getUri() returns the originally requested URI while getRedirectedTo() exposes the final destination.
- How should I upgrade to Escargot 2.0 safely in production?
- Run the queue schema migration before deploying the package bump, because a missing redirected_to column causes runtime SQL errors during a crawl. Audit every implementation of ExceptionSubscriberInterface and update onTransportException() in the same release. Re-validate subscribers, dedupe keys, sitemaps, and exports that previously assumed a single current URI.
- Can Escargot crawl APIs or non-HTML resources?
- Yes, Escargot is designed to crawl any HTTP resource, not just websites. Subscribers decide which requests to follow and how to process responses, so JSON APIs, feeds, and other content types are valid targets. The HTML crawler behavior is one option rather than a hard requirement.
- How do subscribers customize Escargot crawling logic?
- Subscribers let you hook into request filtering, content processing, and exception handling with granular control. You can add custom subscribers, use lazy-loaded tags, and tailor the crawler for SEO audits, sitemap generation, or API scraping. In Laravel, register subscribers through the service container and keep each subscriber small and testable.
- How does Escargot 2.0 handle redirects?
- CrawlUri::getUri() keeps the originally requested URI, while getRedirectedTo() exposes the final destination. The HTML crawler resolves relative links against the final destination, which reduces malformed discovered URLs. Any custom queue adapter must round-trip redirected_to so redirect information survives pause and resume.
- Can long Escargot crawls be paused and resumed?
- Yes, every Escargot instance is tied to a job ID and backed by a queue, so long crawls can stop and continue later. Use a persistent queue such as DoctrineQueue or a custom Laravel adapter instead of InMemoryQueue in production. The job ID is the key you use to resume the same crawl.
- How do I test Escargot in a Laravel test suite?
- Use InMemoryQueue for fast tests and inject fake or stub subscribers to avoid real network calls where possible. Assert on discovered URIs, redirect destinations, and exception handling rather than on live HTTP responses. For integration tests, point the crawler at a local test server or a mocked Symfony HttpClient.
Weaver
How can I help you explore Laravel packages today?