Crawler Teaser Indexer Laravel Package

Product Decisions This Supports

• Search Expansion: Enables indexing of external content (e.g., partner sites, news aggregators) into Solr for unified search results, reducing reliance on manual content ingestion.
• Automated Content Syndication: Supports “headless” content distribution for websites requiring dynamic teasers (e.g., government portals, corporate newsrooms) without direct API access.
• Build vs. Buy: Justifies buying this package over custom development for teams lacking crawling/Solr expertise, given its modular configuration and Symfony/Atoolo integration.
• Upgrade Planning: Version 2.0.0 is primarily a maintenance/refactor release: scheduling command refactored, atoolo/index-bundle adopted, PHP configuration aligned with other Atoolo projects, and tooling standardized. Treat as a compatibility/migration checkpoint rather than as a feature release.
• Roadmap Priorities:
  • Phase 1: Validate 2.0.0 compatibility in an existing Solr-backed search integration and confirm scheduler behavior after the scheduling-command refactor.
  • Phase 2: Extend to support structured data extraction (e.g., schema.org) for richer search results.
  • Phase 3: Add analytics to track crawler performance (e.g., success rates, Solr index growth).

When to Consider This Package

• Adopt if:
  • Your stack includes Symfony/Atoolo + Solr and you need to index unstructured external content (e.g., partner sites, forums).
  • You can adopt the atoolo/index-bundle dependency introduced/refactored in 2.0.0, or are already aligned with the Atoolo package ecosystem.
  • You lack resources to build a custom crawler but require scheduled, configurable scraping (e.g., nightly updates).
  • Your use case aligns with teaser-based indexing (titles, intros, dates) rather than full-page archiving.
• Look elsewhere if:
  • You need JavaScript-heavy sites (this crawler is PHP-based; consider Puppeteer/Playwright).
  • Your target sites enforce aggressive anti-scraping (e.g., Cloudflare challenges).
  • You require real-time indexing (this is cron-based; consider a queue system like RabbitMQ).
  • You are pinned to older Atoolo/Symfony dependencies and cannot absorb a major-version upgrade or the atoolo/index-bundle integration change without migration effort.
  • Your budget allows for enterprise tools (e.g., Scrapy, Apify) with better scalability.

How to Pitch It (Stakeholders)

For Executives: “This package automates the indexing of external content into our Solr search engine, eliminating manual data entry and expanding our search surface without additional dev resources. Version 2.0.0 is a maintenance and integration-alignment release—no new user-facing features are highlighted—so the business value remains the same: broader search coverage with low licensing risk. The main decision is whether to schedule an upgrade to stay aligned with the Atoolo ecosystem; engineering should first confirm scheduler and dependency compatibility.”

For Engineering: “A lightweight, PHP-based crawler for Solr that handles config-driven scraping, Solr schema compatibility for teasers, retry logic, and Symfony/Atoolo integration. 2.0.0 refactors the scheduling command, adopts/uses atoolo/index-bundle, aligns PHP version configuration with other Atoolo projects, and standardizes tooling on php-cs-fixer/phplint. No new features or deprecations are called out in the release notes, but this is a major version: validate dependency resolution, scheduler command behavior, and deployment scripts before upgrading. Recommend pairing with the Atoolo scheduler stack in production.”