Laravel Site Search Laravel Package

Technical Evaluation

Architecture Fit

• Search Functionality: Ideal for Laravel applications requiring full-text search without external dependencies (SQLite FTS5) or with optional Meilisearch for advanced features (typo tolerance, faceting, etc.).
• Crawling Capabilities: Enables site-wide indexing (e.g., blogs, documentation, or e-commerce product pages) by crawling routes dynamically.
• Customization: Supports fine-grained control over what gets indexed (e.g., exclude routes, filter content, or prioritize specific models).
• Decoupled Design: Works alongside existing Laravel features (e.g., Eloquent models, Blade templates) without forcing a monolithic architecture.

Integration Feasibility

• Laravel Native: Built for Laravel (v9+), leveraging Service Providers, Artisan commands, and Queue Workers for crawling.
• Database Agnostic: Defaults to SQLite but supports PostgreSQL/MySQL via FTS5 extensions or Meilisearch for scalability.
• API-Driven: Exposes a clean search API (Search::query()) for frontend integration (e.g., Livewire, Inertia, or JavaScript).
• Event-Based: Triggers CrawlingStarted, PageCrawled, and CrawlingFinished events for extensibility (e.g., logging, analytics).

Technical Risk

• Crawling Complexity:
  • Dynamic Routes: May struggle with SPA-like routes (e.g., React/Vue) or authenticated content unless explicitly configured.
  • Rate Limiting: Concurrent crawling could trigger server throttling or IP bans if not rate-limited (configurable via CrawlOptions).
• Index Freshness:
  • Scheduled Crawls: Requires Artisan scheduling (schedule:run) or queue workers to avoid stale indexes.
  • Delta Updates: No built-in incremental crawling (must re-crawl or implement custom logic).
• Meilisearch Dependency:
  • Adds external service complexity (hosting, sync, cost) if opting out of SQLite.
• Performance Overhead:
  • Initial crawl may block requests if not queued; memory-intensive for large sites.

Key Questions

1. Scope of Search:
   • Will this replace an existing search (e.g., Algolia, Elasticsearch) or supplement it?
   • Are there excluded routes/models (e.g., admin pages, API endpoints)?
2. Crawling Strategy:
   • How often will the site be re-crawled? (Daily? On-demand?)
   • Will authenticated routes need to be crawled? (Requires session handling.)
3. Database Choice:
   • Is SQLite sufficient, or are Meilisearch’s advanced features (e.g., typo tolerance) critical?
   • What’s the expected index size? (SQLite may hit limits for >100K pages.)
4. Scalability:
   • Will the app need to scale crawls horizontally (e.g., distributed workers)?
   • Are there API rate limits (e.g., for external services like Meilisearch)?
5. Monitoring:
   • How will crawl success/failure be monitored? (Events + logging?)
   • Are there alerts for broken links or crawl errors?

Integration Approach

Stack Fit

• Laravel Ecosystem:
  • Service Provider: Registers SearchServiceProvider with bindings for Search and Crawler.
  • Artisan Commands: spatie:crawl for manual triggers; spatie:crawl-schedule for automated runs.
  • Queue Workers: Uses Laravel Queues (spatie:crawl-worker) for concurrent crawling.
  • Events: Integrates with Laravel’s event system for post-crawl actions.
• Frontend:
  • Livewire/Inertia: Directly call Search::query() in components.
  • JavaScript: Fetch results via API routes (e.g., GET /search?q=term).
• Database:
  • SQLite: Zero config; stored in storage/app/search.sqlite.
  • Meilisearch: Requires PHP client (meilisearch/meilisearch-php) and config (MEILISEARCH_URL).

Migration Path

1. Pilot Phase:
   • Start with SQLite for a subset of routes (e.g., blog posts).
   • Test crawling non-authenticated routes first.
2. Gradual Rollout:
   • Replace legacy search (e.g., DB::where() queries) with spatie/laravel-site-search.
   • Use feature flags to toggle between old/new search.
3. Meilisearch Transition (if needed):
   • Set up Meilisearch instance (self-hosted or cloud).
   • Configure Search facade to use Meilisearch via .env:

     SEARCH_DRIVER=meilisearch
     MEILISEARCH_URL=http://meilisearch:7700
     

   • Migrate existing SQLite index to Meilisearch using its import API.

Compatibility

• Laravel Versions: Officially supports v9+; test for v10/11 compatibility.
• PHP Versions: Requires PHP 8.0+ (FTS5 and Meilisearch client dependencies).
• Dependencies:
  • SQLite: Built-in; no extra setup.
  • Meilisearch: Requires meilisearch/meilisearch-php (≥v0.25).
  • Crawler: Uses symfony/dom-crawler and guzzlehttp/guzzle (handled via Composer).
• Route Conflicts: Ensure crawl routes (e.g., /search) don’t clash with app routes.

Sequencing

1. Setup:
   • Install package: composer require spatie/laravel-site-search.
   • Publish config: php artisan vendor:publish --provider="Spatie\SiteSearch\SiteSearchServiceProvider".
   • Configure config/site-search.php (routes, excluded paths, drivers).
2. Crawl Configuration:
   • Define CrawlMap in app/Providers/SiteSearchServiceProvider.php:

     public function boot(): void
     {
         SiteSearch::crawlMap(function (CrawlMap $map) {
             $map->route('/posts/{post}', Post::class);
             $map->route('/docs/*', DocumentationPage::class);
         });
     }
     

3. Initial Crawl:
   • Run manually: php artisan spatie:crawl.
   • Or schedule via app/Console/Kernel.php:

     $schedule->command('spatie:crawl')->daily();
     

4. Search Integration:
   • Add search form to Blade:

     <form action="/search" method="GET">
         <input type="text" name="q">
     </form>
     

   • Create API route (routes/web.php):

     Route::get('/search', [SearchController::class, 'index']);
     

   • Implement controller:

     public function index(Request $request)
     {
         $results = Search::query($request->q)->get();
         return view('search.results', compact('results'));
     }
     

5. Monitoring:
   • Log crawl events (e.g., CrawlingStarted).
   • Set up health checks for Meilisearch (if used).

Operational Impact

Maintenance

• Crawl Management:
  • Scheduled Tasks: Monitor spatie:crawl jobs in Laravel Horizon or queue workers.
  • Error Handling: Implement retries for failed crawls (e.g., transient HTTP errors).
  • Index Pruning: SQLite may need vacuum maintenance for large indexes.
• Configuration Drift:
  • Track changes to CrawlMap and excluded routes in feature flags or migration docs.
• Dependency Updates:
  • Watch for Meilisearch client or PHP version compatibility breaks.

Support

• Troubleshooting:
  • Crawl Failures: Check storage/logs/laravel.log for CrawlFailed events.
  • Search Issues: Verify Search::query() returns expected results; test with Search::getIndexedPages().
  • Meilisearch: Use its admin API to debug sync issues.
• Documentation:
  • Internal Runbooks: Document crawl schedules, excluded routes, and fallback procedures.
  • Frontend Guides: Clarify search result formatting (e.g., highlighting, pagination).
• Vendor Support:
  • Spatie offers GitHub issues and commercial support (via spatie.be).

Scaling

• Crawl Performance:
  • Concurrency: Adjust MAX_CONCURRENT_REQUESTS in config (default: 5).
  • Queue Workers: Scale horizontally for large sites (e.g., Kubernetes pods).
• Database Scaling: