Vektor Laravel Package

Getting Started

Minimal Steps

1. Installation

   composer require centamiv/vektor
   

   Ensure your php.ini has extension=php_vektor.so enabled (if using native extensions).

2. Basic Initialization

   use Centamiv\Vektor\VectorDB;
   
   $vectorDB = new VectorDB(storage_path('app/vectors'));
   

   • Store vectors in a dedicated directory (e.g., storage/app/vectors).

3. First Use Case: Vector Operations

   // Create and manipulate vectors
   $vector = new \Centamiv\Vektor\Vector([1.0, 2.0, 3.0]);
   $normalized = $vector->normalize();
   
   // Store/retrieve vectors
   $vectorDB->save('user_123', $vector);
   $retrieved = $vectorDB->get('user_123');
   

4. Laravel Integration Bind the service in AppServiceProvider:

   public function register()
   {
       $this->app->singleton(VectorDB::class, function () {
           return new VectorDB(storage_path('app/vectors'));
       });
   }
   

Implementation Patterns

1. Vector Storage with Eloquent

Pattern: Sync vectors with Eloquent models using accessors/mutators.

// Model: Post.php
protected $vector;

public function getVectorAttribute()
{
    return $this->vector ?? $this->generateVector();
}

public function setVectorAttribute($value)
{
    $this->vector = $value;
    app(VectorDB::class)->save($this->id, $value);
}

// Generate vector (e.g., from text)
protected function generateVector()
{
    return $this->textToVector($this->content);
}

2. Similarity Search Workflow

Pattern: Combine Laravel queries with vector search.

// Find similar posts by vector
public function findSimilarPosts($queryVector, $limit = 5)
{
    $similarIds = app(VectorDB::class)
        ->search('posts', $queryVector, $limit)
        ->getIds();

    return Post::whereIn('id', $similarIds)->get();
}

3. Batch Processing

Pattern: Use Laravel Queues for bulk operations.

// Job: ProcessVectorsJob.php
public function handle()
{
    $vectors = $this->fetchVectorsFromAPI();
    app(VectorDB::class)->insertBatch('collection', $vectors);
}

4. Hybrid Search with Metadata

Pattern: Filter vectors by metadata before similarity search.

// Pre-filter by category, then search vectors
public function searchByCategory($category, $queryVector)
{
    $candidates = Post::where('category', $category)->pluck('id');
    $similar = app(VectorDB::class)
        ->searchSubset($candidates, $queryVector, 10);
    return Post::find($similar);
}

5. Caching Layer

Pattern: Cache frequent vector queries.

public function cachedSearch($queryVector)
{
    return Cache::remember(
        "vector_search_{$queryVector->hash()}",
        now()->addHours(1),
        fn() => app(VectorDB::class)->search('posts', $queryVector, 10)
    );
}

Gotchas and Tips

Pitfalls

1. Binary Storage Quirks

   • Vectors are stored in binary format. Avoid mixing with JSON/Eloquent serialization unless explicitly supported.
   • Fix: Use json_encode for metadata, but store vectors separately.

2. Dimension Mismatch

   • Operations fail if vectors have mismatched dimensions.
   • Fix: Validate dimensions before operations:

     if ($vector1->dimension() !== $vector2->dimension()) {
         throw new \InvalidArgumentException("Dimension mismatch");
     }
     

3. No Built-in Indexing

   • Without indexing, search performance degrades with dataset size.
   • Workaround: Implement HNSW or KD-trees manually or use a hybrid approach with Redis.

4. Thread Safety

   • Not thread-safe by default. Concurrent writes may corrupt data.
   • Fix: Use Laravel’s queue system or file locking:

     if (file_lock($filePath)) {
         // Critical section
         file_unlock($filePath);
     }
     

5. Memory Leaks

   • Large vectors held in global state can bloat memory.
   • Tip: Use unset() after operations or leverage Laravel’s dependency injection.

Debugging Tips

• Log Vector Operations

  \Log::debug('Vector operation', [
      'vector' => $vector->toArray(),
      'result' => $result->toArray(),
  ]);
  

• Validate Binary Data Use hexdump or xxd to inspect stored binary files if queries fail.

• Benchmark Early Test with your actual dataset size:

  $start = microtime(true);
  $results = $vectorDB->search('collection', $queryVector, 100);
  \Log::info("Search time: " . (microtime(true) - $start) . "s");
  

Extension Points

1. Custom Distance Metrics Extend the Centamiv\Vektor\Distance trait to add metrics like Jaccard similarity:

   class CustomDistance extends \Centamiv\Vektor\Distance
   {
       public static function jaccard($a, $b)
       {
           // Implement custom logic
       }
   }
   

2. Storage Backends Replace the default filesystem storage with a custom backend (e.g., SQLite):

   class SQLiteVectorStore implements \Centamiv\Vektor\StorageInterface
   {
       // Implement save/get/delete methods
   }
   

3. Laravel Scout Driver Create a custom Scout driver for vector search:

   class VektorScoutEngine extends Engine
   {
       public function search($query)
       {
           return app(VectorDB::class)->search('scout', $query->vector, $query->limit);
       }
   }
   

Configuration Quirks

• Storage Path Permissions Ensure the storage directory is writable by the web server user (e.g., chown -R www-data:www-data storage/app/vectors).

• PHP Extensions If using native extensions, enable them in php.ini:

  extension=vektor
  

• Laravel Cache Integration For hybrid caching, configure config/cache.php to use file or database as a fallback.

Performance Tips

• Batch Writes Use insertBatch instead of individual save calls for bulk operations.

• Pre-filter Data Reduce search space by filtering metadata in Laravel before passing to Vektor.

• Monitor Disk Usage Binary storage can grow rapidly. Set up Laravel’s filesystem disk monitoring:

  $diskUsage = Storage::disk('vectors')->diskUsage();