Crawler Teaser Indexer Laravel Package

Technical Evaluation

Architecture Fit

• Symfony/Laravel Compatibility: The package is a Symfony bundle, but its core functionality (crawler logic, Solr indexing) is PHP-agnostic and can be adapted for Laravel via:
  • Service Container Integration: Laravel’s IoC container can register the crawler as a service.
  • Console Command Wrapper: The crawler:index command can be replicated in Laravel’s Artisan CLI.
  • Configuration System: Symfony’s YAML/array config can be replaced with Laravel’s .env or config/crawler.php.
• Solr Dependency: Requires Apache Solr (or compatible search engine like Elasticsearch) for indexing. Laravel’s ecosystem (e.g., Scout, Algolia) may need bridging.
• Event-Driven Potential: The crawler’s modular design (URL discovery → extraction → scoring → indexing) lends itself to Laravel events (e.g., CrawlerDiscovered, TeaserIndexed).

Integration Feasibility

• Low Risk for Core Features:
  • Crawling Logic: Reusable via Laravel’s GuzzleHttp or Symfony\Component\DomCrawler.
  • CSS Selectors: Laravel’s FilamentPHP/spatie-array-to-xml or Symfony/CSSSelector can parse selectors.
  • Solr Client: Laravel’s solrphp/solr-php-client or solarium/solarium can replace the bundle’s Solr integration.
• High Risk for Bundled Features:
  • Scheduler: Symfony’s atoolo-scheduler is not natively available in Laravel. Alternatives:
    • Laravel’s spatie/scheduler or laravel-horizon for queues.
    • External cron jobs triggering Laravel’s artisan command.
  • Monolog Logging: Replace with Laravel’s Monolog or Laravel Log channels.

Technical Risk

Key Questions

1. Solr vs. Alternative Search:

   • Is Solr a hard requirement, or can we use Elasticsearch (via laravel-elasticsearch) or Algolia (via Scout)?
   • If Solr is mandatory, how will we containerize it (Docker, managed service)?

2. Cron vs. Laravel Scheduling:

   • Should the crawler run on a fixed schedule (cron) or event-based (e.g., triggered by a new URL submission)?

3. Configuration Management:

   • How will we version-control crawler configurations (e.g., sp_title_css)?
   • Should configurations be database-driven (e.g., Laravel migrations) or file-based?

4. Error Handling & Retries:

   • How will we log failures (e.g., blocked URLs, Solr timeouts)?
   • Should retries be exponential (as in the bundle) or fixed-delay?

5. Scaling:

   • Will the crawler run on a single instance or distributed workers (e.g., Laravel Queues + Horizon)?
   • How will we rate-limit requests to avoid IP bans?

6. Teaser Deduplication:

   • How will we detect duplicate teasers (e.g., same URL with different query params)?
   • Should we use Solr’s built-in deduplication or Laravel’s cache?

7. Testing:

   • How will we mock external sites for unit/integration tests?
   • Should we use PestPHP or PHPUnit with vcr/vcr for HTTP mocking?

Integration Approach

Stack Fit

Migration Path

1. Phase 1: Core Crawler Logic (2-3 weeks)

   • Extract crawler logic into Laravel services:
     • UrlCollector (handles sp_start_urls, sp_link_selector, etc.).
     • TeaserExtractor (handles CSS selectors, OpenGraph parsing).
     • SolrIndexer (abstracts Solr/Elasticsearch calls).
   • Replace Symfony’s Client with Laravel’s Http client.
   • Test: Verify crawling and extraction against a mock site (e.g., laravel-shift/laravel-http-faker).

2. Phase 2: Configuration & CLI (1 week)

   • Replace YAML config with Laravel’s config/crawler.php.
   • Create artisan crawler:index command to trigger the crawler.
   • Test: Validate config parsing and command execution.

3. Phase 3: Scheduling & Scaling (1-2 weeks)

   • Integrate with Laravel’s task scheduler or external cron.
   • Implement queue-based execution for parallel requests (using sp_parallel_requests).
   • Test: Load test with multiple sites (e.g., laravel-shift/laravel-testing).

4. Phase 4: Solr/Elasticsearch Integration (1 week)

   • Replace Solr client with Laravel-compatible library.
   • Add fallback to Elasticsearch if Solr is unavailable.
   • Test: Verify indexing and search queries.

5. Phase 5: Monitoring & Observability (1 week)

   • Add Laravel Horizon for queue monitoring.
   • Integrate Laravel Telescope for debugging crawler logs.
   • Test: Simulate failures (e.g., Solr downtime, blocked URLs).

Compatibility

• PHP 8.2+: Compatible with Laravel 9/10.
• Symfony Dependencies:
  • Replace Symfony\Component\DomCrawler with Laravel’s FilamentPHP/spatie-array-to-xml or native DOMDocument.
  • Replace Symfony\Contracts\HttpClient with Laravel’s Http client.
• Solr/Elasticsearch:
  • Ensure the target search engine supports the same schema (e.g., sp_id, sp_title, sp_introText).

Sequencing

1. Prerequisites:

   • Laravel 9/10 installed.
   • Solr/Elasticsearch instance running (or Algolia/Scout configured).
   • Composer dependencies: guzzlehttp/guzzle, solrphp/solr-php-client, spatie/array-to-xml.

2. Development Order:

   • Step 1: Implement UrlCollector and TeaserExtractor services.
   • Step 2: Build the CrawlerCommand and config system.
   • Step 3: Integrate Solr/Elasticsearch client.
   • Step 4: Add scheduling (cron or Laravel scheduler).
   • Step 5: Implement retry logic and logging.

3. Deployment Order:

   • Alpha: Single-site crawl (manual artisan trigger).
   • Beta: Scheduled crawls with queue workers