Ai Azure Search Store Laravel Package

Getting Started

Minimal Setup

1. Install the Package:

   composer require symfony/ai-azure-search-store
   

2. Configure Azure AI Search:
   • Create an Azure AI Search resource with a vector index.
   • Note your endpoint URL and API key.
3. Basic Laravel Integration:

   // config/services.php
   'azure_search' => [
       'endpoint' => env('AZURE_SEARCH_ENDPOINT'),
       'key' => env('AZURE_SEARCH_KEY'),
       'index_name' => env('AZURE_SEARCH_INDEX', 'vectors'),
   ],
   

4. Register the Store:

   // app/Providers/AppServiceProvider.php
   use Symfony\AI\AzureSearchStore\AzureSearchStore;
   use Symfony\Contracts\HttpClient\HttpClientInterface;
   
   public function register()
   {
       $this->app->singleton(\Symfony\AI\Store\StoreInterface::class, function ($app) {
           return new AzureSearchStore(
               $app->make(HttpClientInterface::class),
               config('services.azure_search.endpoint'),
               config('services.azure_search.key'),
               config('services.azure_search.index_name')
           );
       });
   }
   

5. First Use Case:

   // Store embeddings
   $store->upsert('doc1', [1.2, 3.4, ...], ['metadata' => ['category' => 'tech']]);
   
   // Query similar vectors
   $results = $store->findNearest('query_embedding', 3, ['$filter' => 'metadata/category eq \'tech\'']);
   

Where to Look First

• Azure AI Search Documentation for index design and query syntax.
• Symfony AI Store Interface for expected methods (upsert, findNearest, remove).
• ScopedHttpClient Feature for custom HTTP client configurations (e.g., retries, timeouts).

Implementation Patterns

Core Workflows

1. Embedding Storage and Retrieval (RAG Pipeline)

// Generate embeddings (e.g., with OpenAI)
$embeddings = $embeddingService->generate(['text' => $document]);

// Store with metadata
$store->upsert(
    'doc_id_' . uniqid(),
    $embeddings,
    ['metadata' => ['source' => 'user_guide', 'language' => 'en']]
);

// Retrieve for LLM context
$nearest = $store->findNearest($queryEmbedding, 5, [
    '$filter' => 'metadata/language eq \'en\' AND metadata/source eq \'user_guide\''
]);

2. Hybrid Search (Vector + Keyword)

// Combine vector similarity with metadata filters
$results = $store->findNearest($queryEmbedding, 10, [
    '$filter' => 'metadata/category eq \'electronics\' AND price lt 1000',
    '$select' => 'id,metadata/name,metadata/price' // Project only needed fields
]);

3. Bulk Operations

// Batch insert (Azure Search supports bulk API)
$batch = [];
foreach ($documents as $doc) {
    $batch[] = [
        'id' => $doc['id'],
        'embedding' => $doc['embedding'],
        'metadata' => $doc['metadata']
    ];
}
$store->bulkUpsert($batch);

// Delete by filter
$store->remove(['$filter' => 'metadata/createdDate lt datetime\'2023-01-01T00:00:00\'']);

Integration Tips

• Laravel Caching Layer: Cache frequent queries in Redis to reduce Azure Search costs:

  $cacheKey = "azure_search:{$queryHash}";
  $results = Cache::remember($cacheKey, now()->addHours(1), function () use ($store, $queryEmbedding) {
      return $store->findNearest($queryEmbedding, 3);
  });
  

• Azure Index Optimization:
  • Use HNSW for vector similarity and BM25 for keyword search in the same index.
  • Pre-filter data with metadata to reduce vector search space (e.g., language = 'en').
• Error Handling: Wrap store operations in try-catch to handle Azure throttling (429) or rate limits:

  try {
      $results = $store->findNearest(...);
  } catch (AzureSearchException $e) {
      if ($e->getCode() === 429) {
          sleep(2); // Retry after delay
          retry();
      }
      throw $e;
  }
  

• ScopedHttpClient: Customize HTTP clients for retries, timeouts, or middleware:

  $httpClient = \Symfony\Contracts\HttpClient\HttpClientInterface::create([
      'base_uri' => config('services.azure_search.endpoint'),
      'auth_bearer' => config('services.azure_search.key'),
      'timeout' => 30,
      'max_duration' => 60,
  ]);
  $store = new AzureSearchStore($httpClient, ...);
  

Laravel-Specific Patterns

• Service Container Binding: Bind the store to Laravel’s container with optional configuration:

  $this->app->bind(\Symfony\AI\Store\StoreInterface::class, function ($app) {
      return new AzureSearchStore(
          $app->makeWith(HttpClientInterface::class, [
              'base_uri' => config('services.azure_search.endpoint'),
              'auth_bearer' => config('services.azure_search.key'),
          ]),
          config('services.azure_search.index_name')
      );
  });
  

• Artisan Commands: Create a command to sync local data to Azure:

  use Symfony\AI\AzureSearchStore\AzureSearchStore;
  
  class SyncEmbeddingsCommand extends Command
  {
      protected $signature = 'ai:sync-embeddings';
      protected $description = 'Sync embeddings to Azure Search';
  
      public function handle()
      {
          $store = app(AzureSearchStore::class);
          foreach (Model::all() as $model) {
              $store->upsert($model->id, $model->embedding, $model->metadata);
          }
      }
  }
  

• Event Listeners: Trigger Azure updates on model events (e.g., saved):

  Model::saved(function ($model) {
      $store = app(AzureSearchStore::class);
      $store->upsert($model->id, $model->embedding, $model->metadata);
  });
  

Gotchas and Tips

Pitfalls

1. Index Schema Mismatches:

   • Issue: Azure Search requires a pre-defined schema (e.g., vector field dimensions). Mismatches cause 400 Bad Request errors.
   • Fix: Verify your index has a vector field with the correct dimensions (e.g., 1536 for OpenAI embeddings). Use the Azure Portal to check or update the schema.

2. Filter Syntax Errors:

   • Issue: Incorrect $filter syntax (e.g., metadata/category = 'tech' instead of metadata/category eq 'tech') returns empty results or errors.
   • Fix: Use OData query syntax and validate with the Azure Search API Explorer.

3. Rate Limiting:

   • Issue: Azure Search throttles requests (429 errors) during bulk operations.
   • Fix: Implement exponential backoff in your ScopedHttpClient:

     $httpClient = HttpClient::create([
         'on_options' => function (Options $options) {
             $options->setRetryOptions([
                 'max_retries' => 3,
                 'delay_factor' => 2,
                 'delay_multiplier' => 100,
             ]);
         },
     ]);
     

4. Metadata Field Limits:

   • Issue: Azure Search has a 256-byte limit for field names and 16MB for documents.
   • Fix: Flatten nested metadata or use base64-encoded JSON for complex structures.

5. Vector Field Precision:

   • Issue: Azure Search uses float for vectors, which may lose precision for high-dimensional embeddings (e.g., 1536D).
   • Fix: Normalize embeddings (e.g., L2 normalization) before storage.

6. Laravel Service Container Conflicts:

   • Issue: Symfony’s HttpClientInterface may conflict with Laravel’s HttpClient.