Laravel Site Search Laravel Package

Technical Evaluation

Architecture Fit

• Search Indexing as a Service: The package excels as a self-contained, Laravel-native solution for full-text search, eliminating the need for external APIs (e.g., Algolia, Elasticsearch) while retaining core search functionality. It aligns well with monolithic Laravel apps where search is a secondary feature but still critical (e.g., documentation sites, internal wikis, or e-commerce product catalogs).
• Decoupled from Frontend: The crawler/indexer logic is agnostic to presentation layers, making it ideal for headless CMS setups or APIs where search results are consumed via GraphQL/REST.
• Hybrid Storage: Supports SQLite/MySQL/PostgreSQL (FTS5/tsvector/FULLTEXT) natively and Meilisearch for advanced features (synonyms, custom ranking). This flexibility allows TPMs to optimize based on cost, performance, or scalability needs.

Integration Feasibility

• Laravel 12+ Only: Requires PHP 8.4+, which may necessitate dependency upgrades if the app is on an older stack. However, this aligns with Laravel’s long-term support (LTS) roadmap.
• Database Schema: Adds a site_search_documents table and database-specific FTS infrastructure (e.g., SQLite virtual tables). Minimal schema changes but requires migration testing in staging.
• Crawler Dependencies: Leverages spatie/crawler, which may introduce rate-limiting risks if the site has aggressive robots.txt rules or anti-scraping measures (e.g., Cloudflare). Mitigation: Customize SearchProfile to throttle requests.
• Queue-Driven Crawling: Uses Laravel queues (site-search:crawl), so queue workers must be configured (e.g., Redis, database). Sync mode (--sync) is available for debugging but not production-recommended.

Technical Risk

Key Questions for TPM

1. Use Case Clarity:
   • Is search a primary feature (e.g., public-facing site) or secondary (e.g., internal tool)? This dictates whether Meilisearch (advanced) or database driver (simple) is preferable.
   • What’s the expected scale? (e.g., 10K vs. 1M pages). Meilisearch scales better horizontally.
2. Content Type:
   • Are pages static (HTML) or dynamic (API-generated)? Dynamic content may need custom Indexer logic.
   • Does the site use authentication? The crawler respects robots.txt but may need adjustments for private routes.
3. Operational Constraints:
   • Are external dependencies (Meilisearch) allowed, or must it be database-only?
   • What’s the acceptable crawl frequency? Daily? Real-time? Queue workers must handle the load.
4. Maintenance:
   • Who will monitor crawl failures (# Failed in site-search:list)?
   • How will index updates be triggered** (e.g., post-deployment, on-demand)?

Integration Approach

Stack Fit

• Best For:
  • Laravel 12+ apps with PHP 8.4+.
  • Projects where search is a secondary feature but requires self-hosted control (no Algolia/Elasticsearch costs).
  • Static or server-rendered content (avoid SPAs unless using Puppeteer crawler).
• Avoid For:
  • High-traffic public sites needing sub-100ms search latency (Meilisearch/Elasticsearch perform better).
  • Highly dynamic content (e.g., user-generated posts) without custom Indexer logic.
  • Multi-language sites (basic stemming; consider Meilisearch’s multilingual support).

Migration Path

1. Assessment Phase:
   • Audit current search implementation (if any). Identify crawlable URLs and indexing requirements.
   • Test with ArrayDriver in a staging environment to validate content extraction.
2. Pilot Index:
   • Create a single index for a subset of pages (e.g., /docs).
   • Use --sync to debug and adjust SearchProfile/Indexer.
3. Full Rollout:
   • Migrate to database driver (SQLite/PostgreSQL recommended for simplicity).
   • Set up cron job for regular crawls (e.g., 0 3 * * * php artisan site-search:crawl).
   • For Meilisearch: Deploy instance (Docker/Cloud) and configure driver_class in config/site-search.php.
4. Frontend Integration:
   • Replace existing search logic with Search::onIndex()->query()->get().
   • Style results using hit->highlightedSnippet() and hit->url.

Compatibility

• Database Drivers:
  • SQLite: Zero config; best for small/medium sites. Uses FTS5 (fast, but limited to single DB file).
  • PostgreSQL: Best for large datasets (tsvector + GIN indexes). Requires pg_trgm extension for fuzzy search.
  • MySQL: Works but has FULLTEXT limitations (e.g., no phrase search in some collations).
• Meilisearch: Requires external instance. Offers synonyms, typo tolerance, and custom ranking, but adds operational overhead.
• Customization:
  • Extend DefaultSearchProfile to filter URLs (e.g., exclude /admin).
  • Override DefaultIndexer to extract custom fields (e.g., product price, author name).

Sequencing

1. Pre-requisites:
   • Upgrade Laravel/PHP to 12/8.4+.
   • Set up queue workers (Redis recommended for performance).
2. Core Setup:
   • Install package: composer require spatie/laravel-site-search.
   • Publish config: php artisan vendor:publish --provider="Spatie\SiteSearch\SiteSearchServiceProvider".
   • Configure config/site-search.php (driver, default profile).
3. Index Creation:
   • Create index: php artisan site-search:create-index (name + URL).
   • Test crawl: php artisan site-search:crawl --sync (debug with ArrayDriver).
4. Production Rollout:
   • Switch to database/Meilisearch driver.
   • Schedule crawls (e.g., daily at 3 AM).
   • Monitor site-search:list for failures.
5. Optimization:
   • Tune SearchProfile to limit crawl scope (e.g., /blog/* only).
   • For Meilisearch: Configure ranking rules for better relevance.

Operational Impact

Maintenance

• Crawl Monitoring:
  • Use php artisan site-search:list to track failed URLs and crawl status.
  • Log crawl errors to Laravel logs or a monitoring tool (e.g., Sentry).
• Index Updates:
  • No built-in triggers: Manually re-crawl or integrate with model observers (e.g., reindex when Post is updated).
  • For Meilisearch: Use webhooks to trigger updates externally.
• Driver-Specific Tasks:
  • SQLite: Backup the .sqlite file regularly.
  • PostgreSQL: Monitor pg_stat_activity for long-running FTS queries.
  • Meilisearch: Handle instance scaling (horizontal for large datasets).

Support

• Troubleshooting:
  • **