Ai Surreal Db Store Laravel Package

Getting Started

Minimal Setup

1. Install Dependencies:

   composer require symfony/ai surreal-db-store surreal-db/surrealdb
   

2. Configure SurrealDB Connection: Add to config/services.php:

   'surrealdb' => [
       'dsn' => 'http://user:pass@localhost:8000',
       'namespace' => 'test',
       'database' => 'test',
   ],
   

3. Define a Vector Index (SurrealQL):

   DEFINE INDEX vector_index ON table USING vector HNSW METRIC cosine DIMENSIONS 1536;
   

4. Bind the Store in Laravel:

   // app/Providers/AppServiceProvider.php
   public function register()
   {
       $this->app->bind(\Symfony\AI\Store\VectorStoreInterface::class, function ($app) {
           $client = new \Surreal\Client($app['config']['surrealdb.dsn']);
           $client->signin(['user', 'pass'], ['namespace', 'database']);
           return new \Symfony\AI\Store\SurrealDbStore($client, 'vector_index', 'table');
       });
   }
   

5. First Usage:

   $store = app(\Symfony\AI\Store\VectorStoreInterface::class);
   $results = $store->find($queryEmbedding, limit: 5);
   

First Use Case: Semantic Search

// Store embeddings
$store->add('doc1', $embeddingArray, ['type' => 'article']);

// Query
$results = $store->find($queryEmbedding, limit: 3, filter: ['type' => 'article']);

// Retrieve metadata
foreach ($results as $result) {
    echo $result->getId(); // 'doc1'
    echo $result->getMetadata()['type']; // 'article'
}

Implementation Patterns

Core Workflows

1. CRUD Operations:

   • Add/Update: Use add() or update() with metadata.

     $store->add('user_123', $userEmbedding, ['role' => 'premium']);
     

   • Remove: Use remove() by ID or filter.

     $store->remove('user_123'); // By ID
     $store->removeWhere(['role' => 'premium']); // Bulk remove
     

   • Batch Operations: SurrealDB supports batch inserts via INSERT statements (wrap in a transaction).

2. Query Patterns:

   • Basic Similarity Search:

     $results = $store->find($queryEmbedding, limit: 10);
     

   • Filtered Search (SurrealDB’s strength):

     $results = $store->find($queryEmbedding, filter: [
         'category' => 'tech',
         'published' => true
     ]);
     

   • Hybrid Queries: Combine vector search with SurrealQL:

     // Custom query via SurrealDB client
     $client->query('SELECT * FROM table WHERE vector_similarity(embedding, ?) > 0.8 AND tags CONTAINS ?', [$queryEmbedding, 'ai']);
     

3. RAG Pipeline Integration:

   // Retrieve context for LLM
   $context = $store->find($queryEmbedding, limit: 3);
   $prompt = "Answer based on: " . implode("\n", $context->getContents());
   

Laravel-Specific Patterns

1. Service Layer Abstraction:

   // app/Services/AI/VectorStoreService.php
   class VectorStoreService {
       public function __construct(private VectorStoreInterface $store) {}
   
       public function search(string $query, array $filters = []): array {
           $embedding = $this->generateEmbedding($query);
           return $this->store->find($embedding, filter: $filters);
       }
   }
   

2. Event-Driven Updates:

   // Listen to model events and update vectors
   Model::updated(function ($model) {
       $embedding = $this->generateEmbedding($model->content);
       $store->update($model->id, $embedding, $model->metadata);
   });
   

3. Caching Layer:

   // Cache results for 5 minutes
   $cacheKey = "vector_search:{$queryHash}";
   $results = Cache::remember($cacheKey, 300, function () use ($store, $queryEmbedding) {
       return $store->find($queryEmbedding);
   });
   

SurrealDB-Specific Tips

1. Index Management:

   • Create indexes before bulk inserts:

     DEFINE INDEX idx_name ON table USING vector HNSW METRIC cosine DIMENSIONS 1536;
     

   • Monitor index health with:

     INFO FOR INDEX idx_name;
     

2. Schema Design:

   • Use tables for structured data (e.g., documents with embedding and metadata fields).
   • Example schema:

     CREATE TABLE documents;
     ALTER TABLE documents ADD COLUMN embedding VECTOR DIMENSIONS 1536;
     ALTER TABLE documents ADD COLUMN metadata MAP;
     

3. Connection Pooling:

   • Reuse the SurrealDB client across requests:

     $client = new \Surreal\Client($dsn);
     $client->signin(['user', 'pass'], ['namespace', 'database']);
     // Reuse $client in multiple store operations
     

Gotchas and Tips

Common Pitfalls

1. Connection Handling:

   • Issue: SurrealDB requires explicit signin before queries.

     $client->signin(['user', 'pass'], ['namespace', 'database']); // Must call this!
     

   • Fix: Wrap in a service provider or use Laravel’s booted event.

2. Vector Dimension Mismatch:

   • Issue: DIMENSIONS in the index must match your embeddings.

     -- Wrong: Will cause errors
     DEFINE INDEX idx ON table USING vector HNSW METRIC cosine DIMENSIONS 384; -- But embeddings are 1536D
     

   • Fix: Validate dimensions before index creation.

3. Filter Syntax:

   • Issue: SurrealQL filters use CONTAINS, IN, etc., not SQL syntax.

     // Wrong (SQL-style)
     $store->find($embedding, filter: ['category = "tech"']);
     
     // Correct (SurrealQL)
     $store->find($embedding, filter: ['category' => 'tech']);
     

   • Fix: Use SurrealQL operators (see docs).

4. Rate Limiting:

   • Issue: SurrealDB’s HTTP API may throttle under high load.
   • Fix: Implement exponential backoff in retries:

     try {
         $results = $store->find($embedding);
     } catch (\Surreal\Exception\RateLimitException $e) {
         sleep(2 ** $attempts);
         retry();
     }
     

5. Metadata Serialization:

   • Issue: Complex metadata (e.g., nested arrays) may not serialize correctly.
   • Fix: Flatten metadata or use JSON strings:

     $store->add('id', $embedding, json_encode(['nested' => ['key' => 'value']]));
     

Debugging Tips

1. Query Logging: Enable SurrealDB query logging:

   $client->setOption('log', true);
   

   Or use Laravel’s logging:

   \Log::debug('SurrealDB Query', ['query' => $client->getLastQuery()]);
   

2. Index Verification: Check if an index exists:

   SHOW INDEXES ON table;
   

3. Performance Profiling: Compare raw SurrealQL vs. Symfony AI wrapper:

   // Benchmark raw SurrealQL
   $start = microtime(true);
   $client->query('SELECT * FROM table WHERE vector_similarity(embedding, ?) > 0.8', [$embedding]);
   $time = microtime(true) - $start;
   

4. Common Errors:

   • Invalid vector dimension: Mismatch between index and embedding size.
   • Index not found: Verify index name/case sensitivity.
   • Authentication failed: Check signin credentials/namespace.

Extension Points

1. Custom Distance Metrics: Extend the store to support custom metrics (e.g., dot product):

   class CustomSurrealDbStore extends SurrealDbStore {
       public function __construct(...) {
           parent::__construct(..., 'dot_product');
       }