Ai Surreal Db Store Laravel Package

Getting Started

Minimal Steps

1. Install Dependencies

   composer require symfony/ai surreal/surrealdb symfony/ai-surreal-db-store
   

   Ensure symfony/ai (v0.8+) is installed as the base package.

2. Configure SurrealDB Connection Add to .env:

   SURREAL_DSN="surreal://user:password@host:port/database?namespace=ns&scope=scope"
   SURREAL_INDEX="vector_index_name"  # Predefined MTREE/HNSW index
   

3. Register the Store in Laravel Bind the store in a service provider (e.g., AppServiceProvider):

   use Symfony\AI\Store\SurrealDbStore;
   use Symfony\AI\Store\StoreInterface;
   
   public function register()
   {
       $this->app->singleton(StoreInterface::class, function ($app) {
           return new SurrealDbStore(
               new \Surreal\SurrealClient($app['config']['surrealdb.dsn']),
               $app['config']['surrealdb.index']
           );
       });
   }
   

4. First Use Case: Store/Retrieve Embeddings

   use Symfony\AI\Store\StoreInterface;
   
   public function storeEmbedding(StoreInterface $store)
   {
       $embedding = [0.1, 0.2, ..., 0.768]; // Example 768D vector
       $id = $store->add('doc_id', $embedding, ['metadata' => 'value']);
   
       $results = $store->find('doc_id', $embedding, 5); // Top 5 similar
       return $results->getIds();
   }
   

Where to Look First

• SurrealDB Schema Setup: Define a table and vector index:

  DEFINE TABLE documents SCOPE public;
  DEFINE INDEX vector_index ON TABLE documents FIELDS vector TYPE MTREE DIMENSIONS 768;
  

• Symfony AI Store Docs: Symfony AI Store Interface
• SurrealDB Vector Guide: Vector Search Reference

Implementation Patterns

Core Workflows

1. Embedding Management

   • Insert: Use add() with metadata (e.g., user ID, timestamp).

     $store->add('user_123', $embedding, ['type' => 'article', 'published' => true]);
     

   • Bulk Insert: Batch operations via SurrealClient directly (not exposed by the store).
   • Update: Replace vectors with add() (idempotent by ID).

2. Similarity Search

   • Basic Query: Filter by metadata + vector similarity.

     $results = $store->find('user_123', $queryEmbedding, 3, [
         'type' => 'article',
         'published' => true,
     ]);
     

   • Distance Metric: Defaults to cosine; override via SurrealQL:

     SELECT * FROM documents WHERE vector ANNIHILATE $query_vector TYPE L2;
     

3. Filtering and CRUD

   • Filter Support: Use Symfony AI’s filter parameter (maps to SurrealQL WHERE).

     $store->find($id, $embedding, 5, ['author' => 'John Doe']);
     

   • Delete: Remove by ID or metadata.

     $store->remove('doc_id'); // By ID
     $store->removeMany(['author' => 'John Doe']); // By filter
     

Integration Tips

• Laravel Eloquent: Attach vectors to models via accessors.

  class Document extends Model
  {
      public function getEmbeddingAttribute()
      {
          return $this->store->get($this->id)?->getEmbedding();
      }
  }
  

• Event-Driven Updates: Trigger SurrealDB writes via Laravel events (e.g., ModelCreated).
• Caching Layer: Cache frequent queries in Redis (SurrealDB’s HTTP latency may be high).

  Cache::remember("similar_{$queryId}", now()->addHours(1), fn() =>
      $store->find($id, $embedding, 5)
  );
  

Advanced Patterns

• Hybrid Search: Combine vector + keyword search.

  SELECT * FROM documents
  WHERE vector ANNIHILATE $query_vector
  AND content CONTAINS 'laravel';
  

• Dynamic Indexing: Rebuild indexes for new dimensions (e.g., switching from 384D to 768D).

  REDEFINE INDEX vector_index ON TABLE documents FIELDS vector TYPE MTREE DIMENSIONS 768;
  

• SurrealQL Customization: Extend the store to support raw queries.

  class ExtendedSurrealDbStore extends SurrealDbStore
  {
      public function customQuery(string $query, array $params): array
      {
          return $this->client->query($query, $params)->toArray();
      }
  }
  

Gotchas and Tips

Pitfalls

1. Index Mismatches

   • Error: Dimension mismatch if embedding size ≠ index dimensions.
   • Fix: Verify DIMENSIONS in DEFINE INDEX matches your embeddings (e.g., 768 for OpenAI’s text-embedding-ada-002).
   • Debug: Check SurrealDB logs for INDEX_NOT_FOUND errors.

2. SurrealQL Syntax Quirks

   • ANNIHILATE vs. NEAREST: Use ANNIHILATE for cosine similarity, NEAREST for exact matches.
   • Filter Order: Place vector conditions first for performance:

     -- Faster
     SELECT * FROM documents WHERE vector ANNIHILATE $query_vector AND metadata = 'X';
     
     -- Slower (filters table first)
     SELECT * FROM documents WHERE metadata = 'X' AND vector ANNIHILATE $query_vector;
     

3. Connection Handling

   • Timeouts: SurrealDB’s HTTP API may time out under load. Increase PHP’s default_socket_timeout:

     ini_set('default_socket_timeout', 30);
     

   • Reconnection: Wrap client calls in retries:

     use Symfony\Component\Process\Exception\ProcessFailedException;
     
     try {
         $results = $store->find($id, $embedding);
     } catch (ProcessFailedException $e) {
         if (str_contains($e->getMessage(), 'connection refused')) {
             $this->reconnectSurrealDb();
             retry();
         }
     }
     

4. Metadata Limitations

   • No Nested Filters: SurrealQL’s WHERE doesn’t support nested JSON queries (e.g., metadata.tags = 'ai'). Flatten metadata or use CONTAINS:

     -- Works
     WHERE metadata LIKE '%"tags": "ai"%'
     
     -- Avoid (fails)
     WHERE metadata.tags = 'ai'
     

Debugging Tips

• Enable SurrealDB Logging Add to config/surrealdb.php:

  'logging' => true,
  

  Check logs for failed queries (e.g., ERROR: INDEX_NOT_FOUND).

• Query Inspection Use SurrealClient::query() to log raw SurrealQL:

  $query = $store->client->query('SELECT * FROM documents LIMIT 1');
  \Log::debug('Raw Query:', [$query->getQuery(), $query->getVariables()]);
  

• Performance Profiling Compare query times with microtime():

  $start = microtime(true);
  $results = $store->find($id, $embedding);
  \Log::info('Query Time:', [microtime(true) - $start]);
  

Extension Points

1. Custom Distance Metrics Override the store’s find() to use L2 distance:

   public function findWithL2(string $id, array $embedding, int $limit = 5, array $filter = []): Results
   {
       $query = "SELECT * FROM {$id} WHERE vector NEAREST $embedding TYPE L2 LIMIT {$limit}";
       return new Results($this->client->query($query, $filter)->toArray());
   }
   

2. Batch Operations Add bulk methods to the store:

   public function addMany(array $data): array
   {
       $ids = [];
       foreach ($data as $item) {
           $ids[] = $this->add($item['id'], $item['embedding'], $item['metadata']);
       }
       return $ids;
   }
   

3. Async Writes Use Laravel Queues