Ai Store Laravel Package

Getting Started

Minimal Setup

1. Install the Core Package:

   composer require symfony/ai-store
   

2. Choose a Bridge (e.g., PostgreSQL with pgvector):

   composer require symfony/ai-postgres-store
   

3. Configure the Store in Laravel’s config/services.php:

   'ai_store' => [
       'default' => env('AI_STORE_DRIVER', 'postgres'),
       'stores' => [
           'postgres' => [
               'dsn' => env('DATABASE_URL'),
               'table' => 'vector_documents',
               'embedding_dimension' => 384, // Match your model's output
           ],
       ],
   ],
   

4. Register the Store Factory in a Laravel service provider:

   use Symfony\Component\AI\Store\StoreFactory;
   use Symfony\Component\AI\Store\Bridge\Postgres\PostgresStore;
   
   public function register()
   {
       $this->app->singleton(StoreFactory::class, function ($app) {
           return new StoreFactory(
               $app['config']['services.ai_store.stores']
           );
       });
   }
   

First Use Case: Indexing Documents

use Symfony\Component\AI\Store\StoreFactory;
use Symfony\Component\AI\Store\TextDocument;

$storeFactory = app(StoreFactory::class);
$store = $storeFactory->getStore('postgres');

$documents = [
    new TextDocument('id1', 'Laravel is a PHP framework...', ['source' => 'docs']),
    new TextDocument('id2', 'Symfony AI provides vector stores...', ['source' => 'docs']),
];

$store->add($documents); // Batch add

First Use Case: Querying Vectors

use Symfony\Component\AI\Store\Query\VectorQuery;

$query = new VectorQuery(
    vector: $embedding, // Pre-computed vector (e.g., from symfony/ai-platform)
    limit: 3,
    distance: 'cosine'
);

$results = $store->query($query);
foreach ($results as $result) {
    echo $result->getText(); // Retrieve document text
}

Implementation Patterns

Core Workflow: RAG Pipeline

1. Vectorize Text (using symfony/ai-platform):

   $vectorizer = app(Symfony\Component\AI\Vectorizer::class);
   $embedding = $vectorizer->vectorize('Your query text');
   

2. Query the Store:

   $query = new VectorQuery($embedding, limit: 5);
   $results = $store->query($query);
   

3. Process Results (e.g., pass to an LLM):

   $context = implode("\n---\n", array_map(fn($r) => $r->getText(), $results));
   

Integration with Laravel

Service Container Binding

// app/Providers/AIServiceProvider.php
public function register()
{
    $this->app->bind(\Symfony\Component\AI\Store\StoreInterface::class, function ($app) {
        return $app->make(StoreFactory::class)->getStore(config('ai_store.default'));
    });
}

Artisan Command for Indexing

// app/Console/Commands/IndexDocuments.php
use Symfony\Component\AI\Store\Indexer\Indexer;
use Symfony\Component\AI\Store\Loader\JsonFileLoader;

protected function handle()
{
    $loader = new JsonFileLoader('path/to/documents.json');
    $indexer = new Indexer($this->store, $loader);
    $indexer->index(); // Batch process documents
}

Event-Driven Extensions

// Listen to query events (e.g., log queries)
use Symfony\Component\AI\Store\Event\PreQueryEvent;

public function boot()
{
    event(new PreQueryEvent($query, $store));
}

Batch Processing Patterns

1. Chunking Large Documents:

   use Symfony\Component\AI\Store\Transformer\TextSplitTransformer;
   
   $transformer = new TextSplitTransformer(
       chunkSize: 1000,
       overlap: 200
   );
   $chunks = $transformer->transform($document);
   

2. Parallel Indexing (Laravel Queues):

   // Dispatch a job for each document batch
   IndexDocumentsJob::dispatch($documents)->onQueue('ai-indexing');
   

Hybrid Search (Keyword + Vector)

use Symfony\Component\AI\Store\Query\HybridQuery;

$query = new HybridQuery(
    vector: $embedding,
    text: 'Laravel framework',
    limit: 3
);

$results = $store->query($query);

Gotchas and Tips

Common Pitfalls

1. Vector Dimension Mismatch:

   • Ensure embedding_dimension in config matches your model’s output (e.g., 384 for sentence-transformers/all-MiniLM-L6-v2).
   • Fix: Update the store’s schema or re-vectorize documents.

2. Memory Leaks with Lazy Iterators:

   • Avoid holding references to lazy iterators (e.g., $store->query() results) in long-lived objects.
   • Tip: Use iterator_to_array() if you need to materialize results immediately.

3. Bridge-Specific Quirks:

   • PostgreSQL: Requires pgvector extension. Run:

     CREATE EXTENSION vector;
     

   • Redis: Uses symfony/ai-redis-store but may need Redis 7+ for vector search.
   • SQLite: Limited to ~100MB databases; not for production-scale data.

4. Experimental Features:

   • StoreFactory and Provider abstraction (v0.8+) may change. Pin versions:

     "symfony/ai-store": "^0.8.0"
     

Debugging Tips

1. Enable Logging:

   $store = $storeFactory->getStore('postgres');
   $store->setLogger($this->app->make(Psr\Log\LoggerInterface::class));
   

2. Inspect Queries:
   • Use PreQueryEvent to log raw queries before execution:

     event(new PreQueryEvent($query, $store));
     

3. Validate Vectors:
   • Check for NaN or infinite values in embeddings:

     $vector = $query->getVector();
     if (array_key_exists(0, array_filter($vector, fn($v) => !is_finite($v)))) {
         throw new \RuntimeException('Invalid vector detected');
     }
     

Performance Optimization

1. Batch Size Tuning:

   • Start with 100 documents per batch. Monitor memory usage and adjust.
   • Example:

     $indexer->setBatchSize(50); // For memory-constrained environments
     

2. Indexing Strategies:

   • Cold Start: Use ResetInterface to clear the store before bulk indexing:

     if ($store instanceof \Symfony\Component\AI\Store\ResetInterface) {
         $store->reset();
     }
     

   • Incremental Updates: Track document IDs to avoid re-indexing unchanged content.

3. Query Optimization:

   • Limit distance calculations for large datasets:

     $query->setDistance('cosine'); // Faster than 'euclidean' for high-dimensional vectors
     

   • Use filter in VectorQuery to narrow results:

     $query->setFilter(['source' => 'docs']);
     

Extension Points

1. Custom Bridges:

   • Implement StoreInterface for unsupported databases (e.g., MongoDB Atlas):

     class MongoDbStore implements StoreInterface
     {
         public function add(iterable $documents): void
         {
             // Custom MongoDB logic
         }
         // ...
     }
     

   • Register with StoreFactory:

     $factory->addStore('mongodb', new MongoDbStore($client));
     

2. Custom Transformers:

   • Extend TransformerInterface for domain-specific preprocessing:

     class DomainSpecificTransformer implements TransformerInterface
     {
         public function transform(TextDocument $document): iterable
         {
             // Custom logic (e.g., extract entities before chunking)
         }
     }
     

3. Event Subscribers:

   • Enhance queries dynamically:

     use Symfony\Component\AI\Store\Event\PreQueryEvent;
     
     $subscriber = new class implements EventSubscriberInterface {
         public function onPreQuery(PreQueryEvent $event)
         {
             $event->getQuery()->setLimit(min($event->getQuery()->getLimit(), 10));
         }
     };
     

Laravel-Specific Tips

1. Cache Store Integration:
   • Use symfony/ai-cache-store with Laravel’s cache: