Crawler Teaser Indexer Laravel Package

Technical Evaluation

Architecture Fit

• Symfony/Laravel Compatibility: The package remains a Symfony bundle, but 2.0.0 moves indexing behind atoolo/index-bundle, which is a meaningful architectural change:
  • Indexing abstraction: The teaser indexer no longer owns Solr client logic directly; it depends on the atoolo/index-bundle contract. This improves Laravel fit — we can implement the index-bundle interface against Elasticsearch, Algolia, or Laravel Scout and skip Solr entirely, rather than reimplementing the crawler’s Solr integration.
  • Service Container Integration: Laravel’s IoC container can register the crawler and indexer as services, binding our own indexer implementation to the index-bundle contract.
  • Console Command Wrapper: The 2.0.0 scheduling command refactor means the command surface (signature, options, exit codes) changed. Our Artisan wrapper must be re-derived from the 2.0.0 source, not the 1.x command.
  • Configuration System: Symfony YAML/array config maps to config/crawler.php / .env as before, but index-related settings now likely live in the index-bundle’s config namespace — split accordingly.
• Index Backend Dependency: Solr is no longer necessarily a hard requirement at the crawler layer; the requirement is now “an implementation of the index-bundle interface.” That is the single most valuable change in this release for a Laravel integration.
• Event-Driven Potential: Unchanged — URL discovery → extraction → scoring → indexing still maps cleanly to Laravel events.

Integration Feasibility

• Low Risk for Core Features:
  • Crawling Logic: Reusable via Laravel’s Http client or Symfony\Component\DomCrawler.
  • CSS Selectors: Symfony/CssSelector remains the pragmatic choice; alternatives are weaker for real-world selector support.
  • Indexing: Now adapter-based via atoolo/index-bundle — a Laravel-side ScoutIndexer or SolrIndexer implementing the bundle contract is the recommended path.
• High Risk for Bundled Features:
  • Scheduler: The 2.0.0 scheduling command refactor invalidates any wrapper built against 1.x. Treat as a breaking change and re-verify command signatures before upgrading.
  • Tooling Churn: phpcs → php-cs-fixer, phive → composer for phplint. Development-only, but it signals the project is still settling its toolchain — expect further minor churn in CI conventions.
• Major Version Signal: The 2.0.0 bump plus the index-bundle extraction is a breaking release. No deprecation notices were published alongside it, so we must diff the public command/config surface ourselves.

Technical Risk

Key Questions

1. Index-bundle contract: What is the exact interface atoolo/index-bundle exposes? Can we implement it against Laravel Scout/Algolia without forking?
2. Command surface: What changed in the scheduling command refactor (name, arguments, output, exit codes)? Is there a documented upgrade guide for 2.0.0?
3. Solr hard requirement: With the index-bundle abstraction, is Solr still required in practice, or is any conforming backend acceptable?
4. Scheduling: Fixed cron vs. event-driven (e.g., triggered by URL submission) — and does 2.0.0 change the recommended scheduling model?
5. Configuration management: File-based (config/crawler.php) vs. database-driven; how do we version-control selector configs?
6. Error handling: How are index-bundle failures surfaced (exceptions vs. result objects) for Laravel logging and retries?
7. Scaling: Single instance vs. distributed workers (Queues + Horizon); how do we rate-limit to avoid IP bans?
8. Teaser deduplication: Solr-native vs. Laravel cache — and does the index-bundle expose dedup hooks?
9. Testing: How do we mock external sites and a fake indexer? PestPHP + vcr/vcr remains our proposal.
10. Upgrade timing: Is 2.0.0 stable enough to target, or do we build on 1.x and port?

Integration Approach

Stack Fit

Migration Path

1. Phase 0: 2.0.0 Impact Triage (2-3 days)

   • Diff the scheduling command and configuration surface between 1.x and 2.0.0.
   • Inspect the atoolo/index-bundle contract and decide build-vs-adopt for the indexer adapter.
   • Decision gate: build directly on 2.0.0, or pin 1.x and port later. Recommendation: build on 2.0.0 — the index abstraction is the reason to adopt.

2. Phase 1: Core Crawler Logic (2-3 weeks)

   • Extract UrlCollector, TeaserExtractor as Laravel services.
   • Replace Symfony’s Client with Laravel’s Http client.
   • Test against a mock site (Http::fake() / vcr/vcr).

3. Phase 2: Indexer Adapter (1-2 weeks)

   • Implement the atoolo/index-bundle contract as a Laravel service.
   • Provide two bindings: Solr (parity) and Scout/Elasticsearch (strategic).
   • Test schema parity (sp_id, sp_title, sp_introText).

4. Phase 3: Configuration & CLI (1 week)

   • Replace YAML with config/crawler.php and config/index.php.
   • Build artisan crawler:index against the 2.0.0 command definition.

5. Phase 4: Scheduling & Scaling (1-2 weeks)

   • Wire Laravel’s scheduler or external cron; confirm the 2.0.0 scheduling model aligns.
   • Queue-based parallel execution via Horizon.

6. Phase 5: Monitoring & Observability (1 week)

   • Horizon for queues, Telescope for debugging, alerting on index failures.

Compatibility

• PHP: 2.0.0 aligns PHP version configuration with other atoolo projects — verify the exact floor (likely 8.2+) before locking our runtime.
• Laravel: Compatible with Laravel 9/10 assuming PHP floor is met.
• Symfony Dependencies: DomCrawler/CssSelector retained; HttpClient replaced with Laravel Http.
• Index Backend: Now contract-based via atoolo/index-bundle — the compatibility question shifts from “does Solr match?” to “can we satisfy the indexer interface?”

Sequencing

1. Prerequisites:
   • PHP at the 2.0.0 floor; Laravel 9/10.
   • atoolo/index-bundle reviewed and version-pinned.
   • Target search backend chosen (Solr for parity, Elasticsearch/Scout for strategy).
2. Development Order:
   • Step 1: Triage 2.0.0 breaking surface.
   • Step 2: UrlCollector + TeaserExtractor.
   • Step 3: Indexer adapter against the index-bundle contract.
   • Step 4: CrawlerCommand + split config.
   • Step 5: Scheduling, retries, logging.
3. Deployment Order:
   • Alpha: single-site manual artisan trigger with a stub indexer.
   • Beta: scheduled crawls, queue workers, Solr adapter.
   • GA: dual indexer adapters, Horizon monitoring, dedup.

Operational Impact

Maintenance

• Positive: Extracting atoolo/index-bundle should reduce our maintenance surface — index changes ship independently of crawler changes, and we no longer patch Solr client code inside the crawler.
• Negative: One more dependency to track and version-align. The 2.0.0 release is a major break with no published migration guide in the notes, so upgrade cost lands on us.
• Toolchain: php-cs-fixer and composer-managed phplint are development-only; no runtime impact, but contributor onboarding docs need updating.

Support

• Upstream appears to be a small, single-vendor team (commit authorship is concentrated). Support is community/best-effort — budget for the possibility of maintaining a fork of the command wrapper.
• The absence of deprecation notes for a major release argues for pinning exact versions and diffing releases ourselves before adopting.

Scaling

• Unchanged: single-instance vs. Horizon-distributed workers is our choice. The index-bundle abstraction helps here — a queue-friendly indexer adapter (batch indexing) is easier to build against a clean contract than against Solr-specific code.
• Rate limiting and per-host politeness remain our responsibility at the HTTP client layer.

Failure Modes

Ramp-Up

• Effort: 5-7 weeks for a mid-level Laravel engineer, with Phase 0 triage added versus the prior 4-6 week estimate.
• Skills: Laravel service providers and queues; HTTP client testing; familiarity with Solr/Elasticsearch schema design to build the indexer adapter.
• Learning curve: The crawler logic is approachable; the new atoolo/index-bundle contract is the main unknown and should be timeboxed as a spike in week one.
• Recommendation: Adopt 2.0.0 as the integration target — the index-bundle extraction materially de-risks the Solr dependency that dominated the 1.x risk profile — but gate the upgrade on a documented diff of the scheduling command and indexer interface.