Resource Crawler Bundle Laravel Package

Technical Evaluation

Architecture Fit

• Symfony Bundle Compatibility: The package is a Symfony bundle, making it a natural fit for Laravel applications only if using Laravel Symfony Bridge or a Symfony microkernel. Native Laravel integration is not straightforward due to Symfony’s dependency injection (DI) container differences.
• Crawling Use Case: The bundle’s core functionality (web/FS crawling, URL filtering, and processing) aligns with Laravel’s needs for asset indexing, SEO crawling, or dynamic content generation.
• Database Schema: The provided Doctrine migrations are not Laravel Eloquent-compatible (uses raw SQL). A custom migration or Eloquent model mapping would be required.

Integration Feasibility

• High: If using Symfony components (e.g., Symfony HTTP Client, Lock system), integration is feasible. For pure Laravel, medium due to DI container mismatches.
• Key Dependencies:
  • symfony/http-client (replaceable with Laravel’s Http facade or Guzzle).
  • symfony/lock (replaceable with Laravel’s Cache or Database locks).
  • ext-dom/ext-libxml (already required for Laravel’s HTML parsing).

Technical Risk

• Medium-High:
  • DI Container: Laravel’s container is not compatible with Symfony’s autowiring. Manual service binding or a bridge (e.g., spatie/laravel-symfony-bridge) is required.
  • Database Schema: Raw SQL migrations won’t work with Laravel’s migrations. Requires conversion to Eloquent or raw Schema::create().
  • Task Rollback: The rollbackTask() method assumes Symfony’s process management. Laravel’s queue system (e.g., shouldRequeue()) would need adaptation.
  • Testing: Limited test coverage (only 1 test file) increases risk of edge-case bugs.

Key Questions

1. Is Symfony integration acceptable?
   • If yes, proceed with spatie/laravel-symfony-bridge.
   • If no, evaluate rewriting core logic in Laravel-native components (e.g., using spatie/crawler as a reference).
2. What’s the primary use case?
   • Web crawling (SEO, scraping) → Prioritize symfony/http-client compatibility.
   • Filesystem crawling → May require custom logic for Laravel’s storage system.
3. How will tasks be managed?
   • Symfony’s Lock system vs. Laravel’s queues (failed_jobs table, retry-after).
4. Database strategy:
   • Use Eloquent models or raw migrations? Trade-offs between maintainability and performance.
5. Error handling:
   • How will errored nodes be logged? Laravel’s Log facade vs. Symfony’s Monolog.

Integration Approach

Stack Fit

Migration Path

1. Phase 1: Dependency Isolation

   • Replace symfony/http-client with Laravel’s Http facade.
   • Replace symfony/lock with Cache::lock() or a custom lock service.
   • Install spatie/laravel-symfony-bridge for partial Symfony compatibility.

2. Phase 2: Database Schema

   • Option A: Eloquent Models

     // app/Models/ResourceCrawlerProcess.php
     class ResourceCrawlerProcess extends Model {
         public function nodes() { return $this->hasMany(ResourceCrawlerNode::class); }
     }
     

   • Option B: Raw Migrations

     Schema::create('resource_crawler_processes', function (Blueprint $table) {
         $table->id();
         $table->string('name');
         $table->timestamps();
     });
     

   • Recommendation: Use Eloquent for maintainability unless performance demands raw SQL.

3. Phase 3: Service Binding

   • Bind the crawler service in AppServiceProvider:

     $this->app->bind('resource_crawler.crawler', function ($app) {
         return new \AndrewSvirin\ResourceCrawlerBundle\Crawler\ResourceCrawler(
             // Inject Laravel-compatible dependencies
         );
     });
     

4. Phase 4: Task Management

   • Replace Symfony’s rollbackTask() with Laravel’s queue logic:

     if ($someExceptionCondition) {
         $task->release(60); // Requeue after 60 seconds (Laravel Queue)
     }
     

Compatibility

• High for Web Crawling: Core crawling logic (URL extraction, filtering) is language-agnostic.
• Medium for Filesystem Crawling: Laravel’s Storage facade may need adaptation for path handling.
• Low for Symfony-Specific Features: Locking, process management, and Doctrine integration require workarounds.

Sequencing

1. Prototype Core Logic
   • Test crawling without Symfony dependencies (e.g., use Guzzle + custom queue).
2. Integrate Symfony Components
   • Gradually replace Laravel equivalents with Symfony components (e.g., symfony/lock).
3. Database First
   • Set up schema before service integration to validate data flow.
4. Error Handling
   • Implement logging (Monolog or Laravel’s Log) before production use.

Operational Impact

Maintenance

• Pros:
  • MIT license allows customization.
  • Symfony bundle structure is modular (easy to extend).
• Cons:
  • Dependency Bloat: Pulls in Symfony components (e.g., symfony/http-client) if not replaced.
  • Documentation Gaps: Limited README and no tests increase maintenance risk.
  • Laravel-Specific Quirks:
    • Queue retries vs. Symfony’s task rollback.
    • Eloquent vs. Doctrine ORM differences.

Support

• Community: Low activity (3 stars, no recent commits). Expect limited community support.
• Debugging:
  • Symfony-Specific Errors: Requires familiarity with Symfony’s DI container.
  • Database Issues: Raw SQL migrations may conflict with Laravel’s schema management.
• Workarounds:
  • Use Laravel’s telescope for debugging crawler tasks.
  • Implement custom logging for errored nodes.

Scaling

• Horizontal Scaling:
  • Challenge: Symfony’s Lock system may not scale as well as Laravel’s distributed cache locks.
  • Solution: Use Redis for distributed locking (Cache::lock() with redis driver).
• Performance:
  • Database: Indexes on status and process_id are provided; ensure Laravel’s query builder optimizes them.
  • Memory: Crawling large sites may hit PHP memory limits. Use Laravel’s queue:work with --memory=4G.
• Concurrency:
  • Laravel’s queues handle concurrency better than Symfony’s process management. Consider:

    $task->onQueue('crawlers'); // Dedicated queue for crawler tasks
    

Failure Modes

Ramp-Up

• Learning Curve:
  • Moderate: Requires understanding of:
    • Laravel’s service container vs. Symfony’s autowiring.
    • Eloquent vs. Doctrine ORM.
    • Symfony’s Lock system vs. Laravel’s caching.
  • High: If using spatie/laravel-symfony-bridge, additional setup