Lucene Search Bundle Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require egeloen/lucene-search-bundle
   

   Add the bundle to config/bundles.php:

   return [
       // ...
       Ivory\SearchBundle\IvorySearchBundle::class => ['all' => true],
   ];
   

2. Configure Indexes: Define a YAML config file (e.g., config/lucene_search.yml) to map Doctrine entities to Lucene indexes:

   ivory_search:
       indexers:
           app_product:
               type: ivory_search.indexer.doctrine.orm
               model: App\Entity\Product
               fields:
                   - { name: name, type: text }
                   - { name: description, type: text }
                   - { name: price, type: numeric }
   

3. First Use Case: Inject the IvorySearchIndexManager service and index an entity:

   use Ivory\SearchBundle\IndexManager;
   
   public function __construct(private IndexManager $indexManager) {}
   
   public function indexProduct(Product $product) {
       $this->indexManager->index('app_product', $product);
   }
   

4. Searching: Use the IvorySearchQueryBuilder to query the index:

   $queryBuilder = $this->indexManager->getQueryBuilder('app_product');
   $results = $queryBuilder
       ->createQuery()
       ->where('name', 'apple')
       ->find();
   

Implementation Patterns

Workflows

1. Indexing Workflows:

   • Batch Indexing: Use IndexManager::indexAll() to reindex all entities in a model.
   • Event-Driven Indexing: Listen to Doctrine lifecycle events (e.g., prePersist, preUpdate) to auto-index entities:

     $entityManager->getEventManager()->addEventListener(
         Doctrine\ORM\Events::prePersist,
         new IndexListener($indexManager, 'app_product')
     );
     

2. Search Workflows:

   • Pagination: Use QueryBuilder::setLimit() and setOffset() for paginated results.
   • Facets: Leverage Lucene's faceting for filtering:

     $queryBuilder->addFacet('category', 'category');
     

3. Hybrid Search: Combine Lucene results with Doctrine queries for precision:

   $luceneResults = $queryBuilder->find();
   $doctrineResults = $entityManager->createQueryBuilder()
       ->where('id IN (:ids)')
       ->setParameter('ids', array_column($luceneResults, 'id'))
       ->getQuery()
       ->getResult();
   

Integration Tips

• Symfony Forms: Use IvorySearchType for search form fields:

  $builder->add('q', IvorySearchType::class, [
      'index' => 'app_product',
      'field' => 'name',
  ]);
  

• APIs: Return search results as JSON:

  return $this->json($queryBuilder->find());
  

• Caching: Cache frequent queries using Symfony's cache system:

  $cache = $this->container->get('cache.app');
  $cacheKey = 'lucene_search_results_' . md5($query);
  $results = $cache->get($cacheKey, function() use ($queryBuilder) {
      return $queryBuilder->find();
  });
  

Gotchas and Tips

Pitfalls

1. Index Schema Changes:

   • Modifying fields in lucene_search.yml requires reindexing the entire dataset. Use migrations or scripts to handle this gracefully.
   • Example migration script:

     $indexManager = $this->container->get('ivory_search.index_manager');
     $indexManager->indexAll('app_product');
     

2. Performance:

   • Large Datasets: Avoid indexing unnecessary fields. Use type: ignore for fields that don’t need searching.
   • Memory Limits: Batch processing is critical for large indexes. Use IndexManager::indexAll() with chunking:

     $indexManager->indexAll('app_product', 100); // 100 entities per batch
     

3. Doctrine Sync Issues:

   • If entities are updated outside Doctrine (e.g., via raw SQL), Lucene indexes will be stale. Use custom listeners or cron jobs to reindex periodically.

4. Lucene Version Mismatches:

   • The bundle defaults to Zend_Search_Lucene. Ensure your composer.json and server environment match the expected Lucene version:

     composer require zendframework/zendsearch-lucene:~1.11
     

Debugging

1. Index Validation:

   • Check if an index exists and is populated:

     $index = $indexManager->getIndex('app_product');
     dump($index->getDocumentCount()); // Should return > 0
     

   • List all documents in an index for debugging:

     foreach ($index->getDocuments() as $doc) {
         dump($doc->getData());
     }
     

2. Query Debugging:

   • Enable Lucene query logging by setting the environment variable:

     export IVORY_SEARCH_DEBUG=1
     

   • Use QueryBuilder::getQuery() to inspect the raw Lucene query:

     dump($queryBuilder->getQuery()->getQuery());
     

3. Common Errors:

   • "Index not found": Verify the index name in lucene_search.yml matches the one used in code.
   • "Field not found": Ensure field names in fields match the entity property names exactly (case-sensitive).

Extension Points

1. Custom Indexers: Extend Ivory\SearchBundle\Indexer\IndexerInterface for non-Doctrine data sources (e.g., CSV files):

   class CustomIndexer implements IndexerInterface {
       public function index($id, $data) {
           // Custom indexing logic
       }
   }
   

   Register it in services.yaml:

   services:
       App\Indexer\CustomIndexer:
           tags:
               - { name: ivory_search.indexer, alias: 'custom_index' }
   

2. Custom Query Builders: Extend Ivory\SearchBundle\QueryBuilder\QueryBuilder to add domain-specific methods:

   class ProductQueryBuilder extends QueryBuilder {
       public function priceRange($min, $max) {
           $this->query->addTerm('price', "[$min TO $max]", 'range');
           return $this;
       }
   }
   

   Override the default query builder in services.yaml:

   services:
       App\QueryBuilder\ProductQueryBuilder:
           arguments:
               - '@ivory_search.index_manager'
               - 'app_product'
           tags:
               - { name: ivory_search.query_builder, alias: 'app_product' }
   

3. Post-Processing Results: Use Ivory\SearchBundle\Result\ResultInterface to transform results before returning them:

   $results = $queryBuilder->find();
   $transformed = array_map(function($result) {
       return $this->productTransformer->transform($result->getData());
   }, $results);