Laravel Site Search Laravel Package

Getting Started

Minimal Steps to Begin

1. Installation

   composer require spatie/laravel-site-search
   php artisan vendor:publish --provider="Spatie\SiteSearch\SiteSearchServiceProvider"
   php artisan migrate
   

2. Create an Index

   php artisan site-search:create-index
   

   • Provide a name (e.g., main) and your site's base URL (e.g., https://example.com).

3. Crawl and Index

   php artisan site-search:crawl
   

   • This queues a job to crawl your site and populate the index.

4. Search

   use Spatie\SiteSearch\Search;
   
   $results = Search::onIndex('main')
       ->query('laravel')
       ->get();
   

First Use Case: Quick Search Implementation

• Use the default DefaultSearchProfile and DefaultIndexer for immediate results.
• Render results in a Blade view:

  @foreach($results->hits as $hit)
      <div>
          <a href="{{ $hit->url }}">{{ $hit->title() }}</a>
          <p>{!! $hit->highlightedSnippet() !!}</p>
      </div>
  @endforeach
  

Implementation Patterns

Core Workflow

1. Define Indexes

   • Use site-search:create-index to define multiple indexes (e.g., blog, products).
   • Configure each index with a unique SiteSearchConfig model (stored in site_search_configs table).

2. Customize Crawling

   • Extend DefaultSearchProfile to filter URLs or modify crawler behavior:

     namespace App\Profiles;
     
     use Spatie\SiteSearch\Profiles\DefaultSearchProfile;
     
     class CustomProfile extends DefaultSearchProfile {
         public function shouldCrawl(string $url): bool {
             return str_contains($url, 'blog') && parent::shouldCrawl($url);
         }
     }
     

   • Register the profile in config/site-search.php:

     'profiles' => [
         'blog' => \App\Profiles\CustomProfile::class,
     ],
     

3. Extract Custom Content

   • Create a custom Indexer for specialized content (e.g., e-commerce products):

     namespace App\Indexers;
     
     use Spatie\SiteSearch\Indexers\Indexer;
     use Carbon\Carbon;
     
     class ProductIndexer implements Indexer {
         public function __construct(public $response) {}
     
         public function pageTitle(): ?string {
             return $this->response->data['title'] ?? null;
         }
     
         public function entries(): array {
             return [
                 ['text' => $this->response->data['description']],
                 ['text' => $this->response->data['features']],
             ];
         }
     
         public function dateModified(): ?Carbon {
             return Carbon::parse($this->response->data['updated_at']);
         }
     }
     

   • Attach the indexer to a profile:

     public function useIndexer(string $url, CrawlResponse $response): ?Indexer {
         return new ProductIndexer($response);
     }
     

4. Search with Filters

   • Use the Search facade to filter results:

     $results = Search::onIndex('products')
         ->query('laptop')
         ->filter('price', '<=', 1000)
         ->get();
     

5. Automate Crawling

   • Schedule crawls via Laravel's scheduler (app/Console/Kernel.php):

     protected function schedule(Schedule $schedule) {
         $schedule->command('site-search:crawl')->daily();
     }
     

Integration Tips

• API Endpoints: Expose search via a controller:

  public function search(Request $request) {
      $results = Search::onIndex('main')
          ->query($request->input('q'))
          ->get();
      return response()->json($results);
  }
  

• Dynamic Indexes: Create indexes programmatically:

  use Spatie\SiteSearch\SiteSearchConfig;
  
  $config = SiteSearchConfig::create([
      'name' => 'dynamic-index',
      'url' => 'https://dynamic-site.com',
      'profile_class' => \App\Profiles\CustomProfile::class,
  ]);
  

• Meilisearch: For advanced features (synonyms, typo tolerance), configure in config/site-search.php:

  'drivers' => [
      'meilisearch' => [
          'host' => env('MEILISEARCH_HOST', 'http://localhost:7700'),
      ],
  ],
  

Gotchas and Tips

Pitfalls

1. Crawl Limits

   • Default crawler limits (e.g., max_pages, max_depth) may exclude critical pages.
   • Fix: Adjust limits in config/site-search.php or override in configureCrawler():

     public function configureCrawler(Crawler $crawler) {
         $crawler->setMaxPages(1000);
         $crawler->setMaxDepth(5);
     }
     

2. Dynamic Content

   • JavaScript-rendered content won’t be indexed by default.
   • Fix: Use tools like Puppeteer or Playwright to render pages before crawling.

3. Rate Limiting

   • Aggressive crawling may trigger 429 Too Many Requests.
   • Fix: Configure delays in configureCrawler():

     $crawler->setDelayBetweenRequests(1000); // 1 second
     

4. Database Driver Quirks

   • SQLite: FTS5 may require UNICODE and TOKENIZE configurations for non-English content.
   • MySQL: Boolean mode (FULLTEXT with +, -, ~) requires careful query construction.
   • PostgreSQL: tsvector weights must be explicitly set for ranking.

5. Indexer Conflicts

   • Custom indexers may override default behavior unintentionally.
   • Fix: Test with ArrayDriver first:

     'default_driver' => \Spatie\SiteSearch\Drivers\ArrayDriver::class,
     

Debugging

1. ArrayDriver Logging

   • Enable for real-time debugging:

     'default_driver' => \Spatie\SiteSearch\Drivers\ArrayDriver::class,
     

   • Check logs for crawl/indexing details:

     tail -f storage/logs/laravel.log
     

2. Synchronous Crawls

   • Run without queues for immediate feedback:

     php artisan site-search:crawl --sync
     

3. Failed URLs

   • List failed URLs:

     php artisan site-search:list --failed
     

   • Retry specific URLs:

     php artisan site-search:retry --url="https://example.com/page"
     

Extension Points

1. Custom Drivers

   • Extend Spatie\SiteSearch\Drivers\Driver for alternative storage (e.g., Elasticsearch):

     namespace App\Drivers;
     
     use Spatie\SiteSearch\Drivers\Driver;
     
     class ElasticsearchDriver implements Driver {
         // Implement index(), search(), etc.
     }
     

2. Pre/Post-Crawl Hooks

   • Use Laravel events to act before/after crawling:

     // In EventServiceProvider
     protected $listen = [
         'site-search.crawling' => [\App\Listeners\LogCrawlStart::class],
         'site-search.crawled' => [\App\Listeners\NotifyCrawlEnd::class],
     ];
     

3. Highlighting Customization

   • Override snippet generation for specific fields:

     namespace App\Drivers;
     
     use Spatie\SiteSearch\Drivers\DatabaseDriver;
     
     class CustomDatabaseDriver extends DatabaseDriver {
         protected function generateHighlightedSnippet(string $text, string $query): string {
             // Custom logic
         }
     }
     

4. Robots.txt Overrides

   • Disable robots.txt checks in a profile:

     public function configureCrawler(Crawler $crawler) {
         $crawler->ignoreRobotsTxt();
     }
     

Performance Tips

1. Batch Crawling

   • Use --batch-size to control queue load:

     php artisan site-search:crawl --batch-size=50
     

2. Selective Indexing

   • Exclude non-critical pages in shouldIndex():

     public function shouldIndex(string $url, CrawlResponse $response): bool {
         return !str_contains($url, 'admin') && parent::shouldIndex($url, $response);
     }
     

3. Database Optimization

   • For SQLite, ensure FTS5 is configured with `