Escargot Laravel Package

Technical Evaluation

Architecture Fit

• Symfony Ecosystem Alignment: Escargot leverages Symfony HttpClient, making it a natural fit for Laravel applications (which already use Symfony components like HttpClient, Console, and Process). This reduces friction in integration and ensures compatibility with Laravel’s existing stack.
• Modular Design: The Subscriber pattern allows for granular control over crawling logic (e.g., request filtering, content processing, error handling). This aligns well with Laravel’s service provider and event-driven architecture. 2.0 note: the subscriber contract changed — ExceptionSubscriberInterface::onTransportException() now receives ResponseInterface|null, so the pattern is unchanged but every implementation of that interface is touched.
• Queue-Based Processing: The job ID and queue system enable resumable crawls, which is critical for long-running tasks in Laravel (e.g., background jobs via Queues, Horizon, or Laravel Forge). 2.0 note: the persistence contract expanded — queues must now round-trip redirect destinations (redirected_to), so any custom/Laravel-native queue adapter is a first-class integration surface, not an afterthought.
• Extensibility: Supports custom subscribers, lazy-loaded tags, and exception handling, allowing TPMs to tailor the crawler to specific use cases (e.g., API scraping, Sitemap generation, or SEO audits). 2.0 adds redirect visibility: CrawlUri::getRedirectedTo() exposes the final destination while getUri() retains the originally requested URI — useful for canonical URL selection, sitemap correctness, and de-duplication.

Integration Feasibility

• Laravel Compatibility:
  • HttpClient: Laravel’s Http facade is built on Symfony HttpClient, so Escargot’s core functionality integrates seamlessly.
  • Queues: Laravel’s queue system (database, Redis, etc.) can replace InMemoryQueue or DoctrineQueue with effort — but under 2.0 a replacement adapter must persist and restore redirect information, or redirect semantics silently degrade.
  • Service Container: Escargot can be registered as a Laravel service provider, enabling dependency injection for subscribers and queues.
• Database Integration: DoctrineQueue works with Laravel’s Eloquent/Query Builder, but a custom queue adapter could be built for Laravel’s database connection. 2.0 requires a schema change (nullable redirected_to) on any existing queue table before the new version is usable.
• Artisan/Console: Escargot’s CLI-friendly design maps well to Laravel’s Artisan commands, enabling scheduled crawls via Laravel Scheduler.

Technical Risk

• BREAKING — Subscriber Signature Change (High): ExceptionSubscriberInterface::onTransportException() now accepts ResponseInterface|null. Existing implementations with the old narrower type are contravariance violations and will fail at class load / first resolution — i.e., a deploy-time fatal, not a graceful runtime error. Mitigation: audit and update every subscriber implementation in the same release; grep the codebase for onTransportException and ExceptionSubscriberInterface.
• BREAKING — Queue Schema Migration (High): DoctrineQueue requires a nullable redirected_to column. Deploying 2.0 before migrating means runtime SQL errors mid-crawl, which is the worst possible failure point for a long-running job. Mitigation: schema migration must precede the package bump in the deploy sequence, and any custom persistent queue must be updated to store/restore redirects.
• Behavior Change — Redirect Semantics (Medium): getUri() retains the original requested URI; getRedirectedTo() exposes the final destination. Any subscriber, dedupe key, or sitemap/export that previously assumed a single "current URI" must be re-validated. Expect different crawl graphs and different stored URLs after upgrade.
• Behavior Change — Relative Link Resolution (Medium): The HTML crawler now resolves relative links against the final destination rather than the original URI. This is more correct and should reduce malformed discovered URLs, but it changes which URLs get queued. Baseline crawl output before/after to avoid attributing the delta to your own code changes.
• State Management: Escargot’s job ID persistence relies on the queue implementation. If using InMemoryQueue, crawls won’t survive process restarts. Mitigation: Use DoctrineQueue or a Redis-backed queue in Laravel (both now redirect-aware).
• Rate Limiting: Escargot lacks built-in rate limiting, which could trigger IP bans or throttling. Mitigation: Implement a custom subscriber to enforce delays (e.g., using Symfony HttpClient’s delay()).
• HTML Parsing Dependencies: The HtmlCrawlerSubscriber may require additional libraries (e.g., symfony/dom) for robust HTML processing. Laravel’s str or html helpers could supplement this.
• Scaling Challenges: For large crawls, memory usage (e.g., InMemoryQueue) or database locks (DoctrineQueue) could become bottlenecks. Mitigation: Use LazyQueue with a fast primary queue (e.g., Redis) and persistent fallback (e.g., database) — and confirm both layers carry redirected_to, otherwise LazyQueue fallback will lose redirect state on recovery.

Key Questions

1. Use Case Clarity:
   • Is the crawler for web scraping, Sitemap generation, SEO analysis, or API data extraction? This dictates subscriber choices (e.g., HtmlCrawlerSubscriber vs. custom JSON parsers) — and, post-2.0, whether redirect destinations must be surfaced in output.
2. Upgrade Readiness (new):
   • Do we have a complete inventory of ExceptionSubscriberInterface implementations? Who owns each, and can they all ship in the same release window as the 2.0 bump?
3. Persistence Requirements:
   • Does the crawl need to resume after failures? If yes, DoctrineQueue or a Laravel queue (e.g., Redis) is mandatory — and under 2.0, redirect persistence is a hard precondition, not optional.
4. Data Contract for URLs (new):
   • Do downstream systems key on the requested URL or the final destination? This decides whether we emit getUri(), getRedirectedTo(), or both, and whether existing records need backfilling.
5. Performance Needs:
   • Will crawls run concurrently (e.g., via Laravel Queues)? If so, queue locking and rate limiting must be addressed.
6. Error Handling:
   • How should HTTP errors (4xx/5xx), network failures, and now pre-response transport failures (null response) be retried? The null case is exactly the bug 2.0 fixes — confirm our subscribers handle "no response object" as a legitimate state rather than an anomaly.
7. Monitoring:
   • Are progress tracking or analytics needed? Laravel’s logging or Laravel Nova could integrate with subscriber callbacks.

Integration Approach

Stack Fit

• Laravel Core:
  • HttpClient: Replace Escargot’s default client with Laravel’s Http facade for consistency.
  • Service Container: Bind Escargot and its dependencies (queues, subscribers) as Laravel services.
  • Queues: Use Laravel’s queue system (e.g., database, redis) as the primary queue, with LazyQueue for fallback persistence — verifying redirect state survives both paths.
• Artisan Commands:
  • Create a custom EscargotCommand to trigger crawls via php artisan escargot:crawl.
  • Support job ID persistence in Laravel’s cache or database.
• Database:
  • Use Laravel’s migrations to create/alter the DoctrineQueue table, including the new nullable redirected_to column.
  • Store crawl metadata (e.g., start time, status) in Laravel’s jobs table or a custom table.

Migration Path

0. Upgrade Audit (do first, 2.0-specific):
   • Enumerate all classes implementing ExceptionSubscriberInterface and all queue adapters. Confirm each can be updated for the new ResponseInterface|null signature and redirect persistence.
1. Initial Setup:
   • Install Escargot via Composer (pin explicitly during the transition):

     composer require terminal42/escargot:^2.0 symfony/http-client symfony/dom
     

   • Register a Laravel service provider to bind Escargot components:

     // app/Providers/EscargotServiceProvider.php
     public function register()
     {
         $this->app->singleton(Escargot::class, function ($app) {
             $queue = new LazyQueue(
                 new RedisQueue(), // Primary: Laravel Redis queue
                 new DoctrineQueue($app['db']) // Fallback: Laravel DB (must persist redirected_to)
             );
             return Escargot::createFromJobId(
                 $app['config']['escargot.job_id'],
                 $queue
             );
         });
     }
     

2. Subscriber Integration:
   • Create custom subscribers (e.g., App\Subscribers\DataExtractorSubscriber) implementing SubscriberInterface.
   • Update any exception subscriber to the 2.0 signature and handle a null response (validation check, below):

     // Verify the exact parameter order/types against the installed vendor interface.
     public function onTransportException(
         CrawlUri $crawlUri,
         ResponseInterface|null $response,
         \Throwable $exception
     ): void {
         if ($response === null) {
             // Failure occurred before a response existed (connection reset, DNS, TLS).
             // Retry/backoff logic must live here rather than in response inspection.
         }
     }
     

   • Register subscribers in the service provider:

     $escargot = $this->app->make(Escargot::class);
     $escargot->addSubscriber(new RobotsSubscriber());
     $escargot->addSubscriber(new HtmlCrawlerSubscriber());
     $escargot->addSubscriber(new DataExtractorSubscriber());
     

3. Queue Schema Migration (must precede the 2.0 deploy):

   // database/migrations/xxxx_add_redirected_to_to_escargot_queue.php
   public function up(): void
   {
       Schema::table('escargot_queue', function (Blueprint $table) {
           $table->string('redirected_to')->nullable();
       });
   }
   

   • For large queue tables, use online/instant DDL where the engine supports it, or drain the queue first. A nullable column add is cheap on modern MySQL/Postgres, but plan the window explicitly.
   • If a custom persistent queue adapter is in use, update its store/restore logic to carry redirected_to, then write a round-trip test.
4. Artisan Command:
   • Build a command to start crawls:

     // app/Console/Commands/EscargotCrawl.php
     public function handle()
     {
         $escargot = app(Escargot::class);
         $escargot->crawl();
     }
     

   • Schedule via Laravel’s scheduler (app/Console/Kernel.php):

     protected function schedule(Schedule $schedule)
     {
         $schedule->command('escargot:crawl')->dailyAt('3:00');
     }
     

5. Queue Configuration:
   • Configure Laravel’s queue driver (e.g., Redis) in .env:

     QUEUE_CONNECTION=redis
     

   • Use LazyQueue to combine speed (Redis) and persistence (database):

     $queue = new LazyQueue(
         new RedisQueue(),        // Fast primary queue
         new DoctrineQueue($app['db']) // Persistent fallback (redirect-aware)
     );
     

Compatibility

• Symfony Components: Laravel’s built-in Symfony components (e.g., HttpClient, DomCrawler) ensure seamless integration. The 2.0 exception change tracks Symfony’s own behavior, where transport failures can occur before a response exists — so the new contract is more compatible with Laravel’s HTTP stack, not less.
• Database Abstraction: DoctrineQueue works with Laravel’s Eloquent or Query Builder, but a custom adapter could use Laravel’s DB facade directly — it must implement the new redirect contract.
• Event System: Escargot’s subscribers can dispatch Laravel events (e.g., CrawlStarted, UriProcessed, UriRedirected) for cross-service communication. Adding a redirect-aware event is low cost now that getRedirectedTo() exists.
• Version Pinning: ^1.7 and ^2.0 are not interchangeable at the interface or schema level. Treat the bump as a coordinated release, not a routine patch.

Sequencing

1. Phase 0: Upgrade Triage (2.0 gate)
   • Inventory subscribers and queue adapters; scope signature updates and redirect persistence.
   • Baseline current crawl output (URLs discovered, redirect handling) for before/after comparison.
2. Phase 1: Core Integration
   • Set up Escargot as a Laravel service.
   • Implement a basic crawler with RobotsSubscriber and HtmlCrawlerSubscriber.
   • Test with InMemoryQueue for validation (accepting no restart resilience).
3. Phase 2: Persistence + Schema Migration
   • Ship the redirected_to migration ahead of the package bump.
   • Replace InMemoryQueue with DoctrineQueue or a Laravel queue adapter that persists redirects.
   • Add job ID persistence (e.g., in Laravel cache or database).
4. Phase 3: Subscriber Migration & Customization
   • Update exception subscribers to the ResponseInterface|null signature.
   • Develop custom subscribers for business logic (e.g., data extraction using getRedirectedTo(), rate limiting).
   • Integrate with Laravel’s logging or monitoring (e.g., Laravel Horizon).
5. Phase 4: Scaling
   • Optimize queue performance (LazyQueue with Redis + database), confirming redirect state survives failover.
   • Add parallel processing via Laravel Queues (e.g., dispatch() crawls as jobs).

Operational Impact

Maintenance

• Dependency Updates:
  • Escargot relies on Symfony components (e.g., HttpClient). Laravel’s built-in Symfony packages will handle most updates, but custom subscribers may need adjustments.
  • 2.0 is a major release: budget an explicit upgrade ticket with code + schema work, not a composer update. Keep a pinned ^1.7 branch available for rollback until the new crawl output is validated.
  • Monitor for breaking changes in Symfony 7+ (e.g., HTTP client improvements) — the ResponseInterface|null contract is likely to persist, so it is a one-time cost.
• Queue Management:
  • Laravel’s queue workers (e.g., php artisan queue:work) will handle Escargot’s queue processing. Supervisor or Horizon can manage worker processes.
  • Queue table maintenance: For DoctrineQueue, ensure the database is optimized (e.g., indexes on job_id, and on redirected_to only if queried). Redirect values are unbounded strings — consider a prefix/length policy if the queue table grows large.
  • Schema drift risk: environments where the migration is skipped (staging clones, ephemeral review apps) will fail only when a crawl runs — add a deploy smoke check that starts and drains a tiny crawl.
• Subscriber Lifecycle:
  • Custom subscribers should implement cleanup logic (e.g., releasing resources, persisting partial results) and be idempotent, since LazyQueue failover can replay entries.
  • Redirect-aware subscribers should record both getUri() and getRedirectedTo() to keep historical data consistent across the upgrade boundary.

Support

• Escalation surface: Two new support categories — "crawl stopped after deploy" (schema) and "subscriber fatal / TypeError class error" (signature). Both are identifiable within minutes if logs capture the first exception and the deployed package version is recorded alongside crawl job IDs.
• Ownership: Assign a named owner for the subscriber inventory and a named owner for the queue schema. The 2.0 upgrade fails in two independent places, so a single owner can mask the second failure mode.
• Vendor cadence: Single-maintainer package (@Toflar). Major releases are infrequent but contract-breaking; subscribe to releases and review changelogs before any composer update in CI.
• Documentation debt: Internal runbooks referencing 1.x redirect behavior must be corrected — getUri() is now explicitly the requested URI.

Scaling

• Throughput: Redirect handling adds one small write per URI; at high crawl volume this shifts load onto the queue store. Benchmark queue write latency before scaling workers, particularly on the database fallback.
• Crawl-graph expansion: Resolving relative links against the final destination can increase discovered URLs (correctly so) on redirect-heavy sites. Re-baseline queue depth, crawl duration, and storage growth assumptions post-upgrade.
• LazyQueue layering: Both layers must be redirect-aware. A Redis primary that stores redirects feeding a fallback that drops them produces rare, hard-to-reproduce data inconsistencies — test failover explicitly rather than assuming it.
• Worker concurrency: More concurrent workers amplify duplicate-URI risk without dedupe/locking; redirect chains make dedupe harder because requested and final URLs both exist. Decide the canonical key before scaling out.

Failure Modes

• Missing redirected_to column → SQL error mid-crawl, partial progress, potentially a stuck job. Detection: crawl aborts with a database exception, not an HTTP error. Mitigation: migration-before-bump deploy ordering.
• Un-updated exception subscriber → fatal class/signature error at resolution time; may take down the whole crawl, not just error handling. Detection: error text referencing the interface, appearing immediately after deploy. Mitigation: static analysis / grep gate in CI.
• null response unhandled → the original 2.0 bug class. Subscribers must not assume $response->getStatusCode() exists; a swallowed null path can silently drop retries and under-report failures.
• Custom queue drops redirect state → links resolved against the wrong base, duplicate/orphaned URLs, incorrect canonical data. Detection: URL set drifts from baseline, no exception raised. Mitigation: round-trip test asserting getRedirectedTo() survives store/restore.
• Stale baseline attribution → post-upgrade crawl output differs (more correct links, new URL fields) and is mistaken for a regression. Detection: diff review against the Phase 0 baseline. Mitigation: capture the baseline before the bump.
• LazyQueue fallback divergence → crawls resume with partial redirect knowledge after a primary-queue failure. Detection: intermittent, non-reproducible URL anomalies.

Ramp-Up

• Effort: Small but non-trivial — one migration, one interface update per exception subscriber, one validation pass on URL-writing code, plus redirect round-trip tests. Sequence it as a discrete ticket with rollback, not a background dependency bump.
• Learning curve: Low for existing Escargot users; the only conceptual shift is the requested vs. final URL distinction, which needs a short team briefing because it changes what "the URL" means in stored data.
• Validation gates: (1) migration applied in every environment, (2) no ExceptionSubscriberInterface signature violations, (3) redirect state survives queue failover, (4) crawl output diffed against the pre-upgrade baseline and explained.
• Rollback: Safe if the migration is additive/nullable and 1.7 ignores the column; confirm any data written with redirect awareness is acceptable to the older version before relying on rollback.