Ai Meilisearch Store Laravel Package

Getting Started

Minimal Setup

1. Install Dependencies:

   composer require symfony/ai symfony/ai-meilisearch-store meilisearch/meilisearch-php
   

2. Configure Meilisearch:
   • Set up a Meilisearch instance (Docker/cloud/self-hosted).
   • Define environment variables:

     MEILISEARCH_HOST=http://localhost:7700
     MEILISEARCH_API_KEY=your_master_key
     MEILISEARCH_INDEX=your_vector_index
     

3. Basic Usage:

   use Symfony\Component\AI\Store\MeilisearchStore;
   use Meilisearch\Client;
   
   $client = new Client(env('MEILISEARCH_HOST'), env('MEILISEARCH_API_KEY'));
   $store = new MeilisearchStore($client, env('MEILISEARCH_INDEX'));
   
   // Add a vector embedding
   $store->add([0.1, 0.2, 0.3], ['id' => 1, 'title' => 'Example Document']);
   
   // Hybrid search (keyword + vector)
   $results = $store->search(
       'laravel',
       [0.1, 0.2, 0.3],
       semanticRatio: 0.5 // 50% keyword, 50% vector
   );
   

First Use Case: Semantic Search for FAQs

1. Precompute Embeddings: Use symfony/ai to generate embeddings for FAQ documents (e.g., with a model like sentence-transformers/all-mpnet-base-v2).
2. Index in Meilisearch:

   $faqEmbeddings = [
       ['embedding' => $embedding1, 'metadata' => ['question' => 'How to install Laravel?']],
       ['embedding' => $embedding2, 'metadata' => ['question' => 'What is Eloquent?']],
   ];
   foreach ($faqEmbeddings as $item) {
       $store->add($item['embedding'], $item['metadata']);
   }
   

3. Query with Hybrid Search:

   $userQuery = "Set up Laravel project";
   $userEmbedding = generateEmbedding($userQuery); // Using symfony/ai
   $results = $store->search($userQuery, $userEmbedding, semanticRatio: 0.7);
   

Implementation Patterns

Core Workflows

1. Vector Indexing

• Bulk Operations:

  $store->addMany([
      ['embedding' => $vec1, 'metadata' => ['id' => 1]],
      ['embedding' => $vec2, 'metadata' => ['id' => 2]],
  ]);
  

• Partial Updates: Use Meilisearch’s primary key (id) to update existing vectors:

  $store->add($newEmbedding, ['id' => 1, 'title' => 'Updated Title']);
  

2. Hybrid Search

• Keyword + Vector Queries:

  $results = $store->search(
      'laravel tutorial',
      $userEmbedding,
      semanticRatio: 0.6, // 60% vector, 40% keyword
      limit: 5
  );
  

• Filtering:

  $results = $store->search(
      'laravel',
      $userEmbedding,
      filters: ['category' => 'tutorial', 'published' => true]
  );
  

3. RAG Pipeline Integration

• Retrieve Documents for LLM Prompts:

  $relevantDocs = $store->search(
      'Explain Laravel middleware',
      $queryEmbedding,
      semanticRatio: 0.8,
      limit: 3
  );
  $prompt = "Answer the question based on these documents: " . json_encode($relevantDocs);
  

4. Caching Layer (Laravel-Specific)

• Cache hybrid search results in Redis:

  use Illuminate\Support\Facades\Cache;
  
  $cacheKey = "hybrid_search:{$userQuery}:".implode(',', $userEmbedding);
  $results = Cache::remember($cacheKey, now()->addHours(1), function () use ($store, $userQuery, $userEmbedding) {
      return $store->search($userQuery, $userEmbedding, semanticRatio: 0.5);
  });
  

Laravel Integration Patterns

Service Provider Binding

// app/Providers/AppServiceProvider.php
public function register()
{
    $this->app->singleton(MeilisearchStore::class, function ($app) {
        $client = new Client(
            env('MEILISEARCH_HOST'),
            env('MEILISEARCH_API_KEY')
        );
        return new MeilisearchStore($client, env('MEILISEARCH_INDEX'));
    });
}

Repository Pattern

// app/Repositories/VectorSearchRepository.php
class VectorSearchRepository
{
    public function __construct(private MeilisearchStore $store) {}

    public function findSimilarDocuments(array $embedding, int $limit = 5): array
    {
        return $this->store->search('', $embedding, limit: $limit);
    }

    public function findHybridResults(string $query, array $embedding, float $semanticRatio = 0.5): array
    {
        return $this->store->search($query, $embedding, semanticRatio: $semanticRatio);
    }
}

Artisan Commands for Maintenance

// app/Console/Commands/ReindexVectors.php
class ReindexVectors extends Command
{
    public function handle()
    {
        $store = app(MeilisearchStore::class);
        $store->clear(); // Remove all vectors
        $newVectors = $this->fetchVectorsFromDatabase();
        $store->addMany($newVectors);
    }
}

Gotchas and Tips

Pitfalls

1. Meilisearch Index Mismatch:

   • Issue: Vectors added to one index won’t appear in searches on another.
   • Fix: Ensure MeilisearchStore uses the same index name for add() and search().
   • Debug: Check Meilisearch dashboard (http://localhost:7700) for active indices.

2. Embedding Dimension Mismatch:

   • Issue: Meilisearch rejects vectors with incorrect dimensions (e.g., 768 vs. 384).
   • Fix: Validate embedding dimensions before indexing:

     $expectedDim = 384;
     if (count($embedding) !== $expectedDim) {
         throw new \InvalidArgumentException("Embedding must have {$expectedDim} dimensions.");
     }
     

3. Hybrid Search Semantic Ratio:

   • Issue: semanticRatio of 0 or 1 disables hybrid search (pure keyword or vector).
   • Tip: Start with 0.5 and adjust based on relevance feedback.

4. Rate Limiting:

   • Issue: Meilisearch may throttle requests during bulk operations.
   • Fix: Use async processing or batch adds:

     $batchSize = 100;
     foreach (array_chunk($vectors, $batchSize) as $batch) {
         $store->addMany($batch);
         sleep(1); // Throttle
     }
     

5. Metadata Filtering:

   • Issue: Filters require filterableAttributes to be set in Meilisearch.
   • Fix: Configure the index schema:

     $client->index('your_index')->updateSchema([
         'filterableAttributes' => ['category', 'published'],
     ]);
     

Debugging Tips

1. Meilisearch Logs: Enable debug logs for Meilisearch:

   MEILI_MASTER_KEY=masterKey
   MEILI_LOG_LEVEL=debug
   

   View logs at http://localhost:7700/logs.

2. Query Validation: Use Meilisearch’s API explorer to test queries:

   curl -X GET "http://localhost:7700/indexes/your_index/search?q=laravel&semanticRatio=0.5"
   

3. Laravel Logging: Log raw queries for debugging:

   $query = $store->search($queryText, $embedding, semanticRatio: 0.5);
   \Log::debug('Meilisearch query', ['query' => $queryText, 'embedding' => $embedding]);
   

Extension Points

1. Custom Scoring: Extend the search method to include Meilisearch’s rankingRules:

   $results = $store->search(
       $query,
       $embedding,
       rankingRules: [
           'typoTolerance' => 'lenient',
           'words' => ['title' => 5], // Boost title matches
       ]
   );
   

2. Laravel Events: Trigger events for vector