Doctrine Orm Batcher Laravel Package

Getting Started

First Steps

1. Installation Add the package via Composer:

   composer require setono/doctrine-orm-batcher
   

   Ensure your Laravel app uses Doctrine ORM (e.g., via ocramius/doctrine-laravel).

2. Basic Setup Register the batcher service in Laravel’s config/services.php:

   'doctrine-batcher' => [
       'batch_size' => 50, // Default batch size
       'flush_after' => true, // Auto-flush after each batch
   ],
   

   Bind the Batcher class in a service provider:

   $this->app->bind(\Setono\DoctrineORMBatcher\Batcher::class, function ($app) {
       return new \Setono\DoctrineORMBatcher\Batcher(
           $app['doctrine.orm.entity_manager'],
           $app['config']['services.doctrine-batcher']
       );
   });
   

3. First Use Case: Bulk Insert Inject the Batcher into a service and use it to insert entities in batches:

   use Setono\DoctrineORMBatcher\Batcher;
   
   public function __construct(private Batcher $batcher) {}
   
   public function bulkInsertUsers(array $usersData): void
   {
       $batch = $this->batcher->getBatch();
       foreach ($usersData as $data) {
           $user = new User();
           $user->setName($data['name']);
           $user->setEmail($data['email']);
           $batch->addEntity($user);
       }
       $this->batcher->flush();
   }
   

Implementation Patterns

Common Workflows

1. Batched CRUD Operations

   • Insert: Use getBatch() to start a batch, add entities with addEntity(), then flush().
   • Update: Fetch entities in batches (e.g., via EntityRepository::findBy()), modify, and batch-update:

     $batch = $this->batcher->getBatch();
     $users = $userRepo->findBy(['active' => false], null, 100);
     foreach ($users as $user) {
         $user->setActive(true);
         $batch->addEntity($user);
     }
     $this->batcher->flush();
     

   • Delete: Use addDelete() for bulk deletes:

     $batch = $this->batcher->getBatch();
     $batch->addDelete($userRepo->findBy(['deleted' => true]));
     $this->batcher->flush();
     

2. Transaction Management Wrap batches in transactions for atomicity:

   $entityManager = $this->batcher->getEntityManager();
   $entityManager->beginTransaction();
   try {
       $this->batcher->flush();
       $entityManager->commit();
   } catch (\Exception $e) {
       $entityManager->rollback();
       throw $e;
   }
   

3. Dynamic Batch Sizing Adjust batch size per operation:

   $this->batcher->setBatchSize(200); // Override default
   

4. Integration with Queues Offload batch processing to queues (e.g., Laravel Queues) to avoid timeouts:

   dispatch(new ProcessBatchJob($usersData))->onQueue('batch-processing');
   

Integration Tips

• Doctrine Events: Listen to onFlush or postFlush to log batch operations:

  $eventManager->addEventListener(
      \Doctrine\ORM\Events::postFlush,
      [$this, 'logBatchOperations']
  );
  

• Custom Batch Callbacks: Extend Batcher to add pre/post-flush logic:

  $this->batcher->setPostFlushCallback(function () {
      Log::info('Batch processed successfully');
  });
  

• Laravel Eloquent: Use DoctrineExtensions (e.g., beberlei/DoctrineExtensions) to bridge Eloquent and Doctrine entities if needed.

Gotchas and Tips

Pitfalls

1. Memory Leaks

   • Issue: Large batches may exhaust memory. Monitor memory_get_usage().
   • Fix: Use smaller batches (e.g., 50–200) or stream data from a file/database cursor.

2. Transaction Timeouts

   • Issue: Long-running transactions may hit database timeouts.
   • Fix: Commit/rollback explicitly or use flushAfter to auto-flush smaller batches.

3. Entity State Conflicts

   • Issue: Detached entities or stale references can cause EntityNotFoundException.
   • Fix: Ensure entities are managed by the same EntityManager or refresh them:

     $entityManager->refresh($entity);
     

4. Batch Ordering

   • Issue: Batches may not execute in the expected order if flushed asynchronously.
   • Fix: Use transactions or queue jobs with strict ordering.

Debugging

• Enable Doctrine Logging:

  $entityManager->getConnection()->getConfiguration()->setSQLLogger(new \Doctrine\DBAL\Logging\EchoSQLLogger());
  

• Check Batch Contents: Inspect the batch before flushing:

  $batch = $this->batcher->getBatch();
  dump($batch->getEntities()); // Debug added entities
  

Configuration Quirks

• flushAfter Behavior: Setting flush_after: true auto-flushes after each batch, which may impact performance for large datasets. Disable it for fine-grained control.
• Batch Size Limits: Some databases (e.g., MySQL) have max_allowed_packet limits. Test with your batch size.

Extension Points

1. Custom Batchers: Extend \Setono\DoctrineORMBatcher\Batcher to add domain-specific logic:

   class CustomBatcher extends Batcher {
       public function addCustomEntity(Entity $entity) {
           $this->batch->addEntity($entity);
           // Add custom validation
       }
   }
   

2. Event Subscribers: Hook into Doctrine events to modify batch behavior:

   $eventManager->addEventSubscriber(new class implements EventSubscriber {
       public function getSubscribedEvents() {
           return [Events::onFlush];
       }
       public function onFlush(OnFlushEventArgs $args) {
           // Modify batch entities here
       }
   });
   

3. Laravel Service Provider: Bind custom batchers or decorate the default Batcher:

   $this->app->decorate(
       \Setono\DoctrineORMBatcher\Batcher::class,
       function ($batcher, $app) {
           return new CustomBatcher($batcher->getEntityManager());
       }
   );