Data Import Bundle Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require ddeboer/data-import-bundle
   

   Add to AppKernel.php:

   new Ddeboer\DataImportBundle\DdeboerDataImportBundle(),
   

2. First Use Case: Import a CSV file into a Doctrine entity. Create a DataImport class extending Ddeboer\DataImport\Import:

   use Ddeboer\DataImport\Import\ImportInterface;
   use Ddeboer\DataImport\Reader\CsvReader;
   use Ddeboer\DataImport\Writer\DatabaseWriter;
   use App\Entity\User;
   
   class UserImport implements ImportInterface
   {
       public function getReader(): CsvReader
       {
           return new CsvReader('path/to/users.csv');
       }
   
       public function getWriter(): DatabaseWriter
       {
           return new DatabaseWriter(User::class);
       }
   
       public function getFields(): array
       {
           return [
               'username' => 'username',
               'email'    => 'email',
               'role'     => 'role',
           ];
       }
   }
   

3. Run the Import:

   $import = new UserImport();
   $import->run();
   

Where to Look First

• Bundle Docs: GitHub README
• Library Docs: ddeboer/data-import
• Symfony Integration: Check DdeboerDataImportBundle for Symfony-specific features (e.g., event dispatching).

Implementation Patterns

Common Workflows

1. CSV to Database: Use CsvReader + DatabaseWriter for bulk inserts/updates. Example:

   $writer = new DatabaseWriter(User::class, [
       'batch_size' => 50, // Optimize for performance
   ]);
   

2. Custom Mappings: Override getFields() to map CSV columns to entity properties or custom logic:

   public function getFields(): array
   {
       return [
           'username' => function ($value) {
               return strtolower(trim($value));
           },
           'email' => 'email',
       ];
   }
   

3. Validation: Use Validator to enforce rules:

   use Ddeboer\DataImport\Validator\Validator;
   
   $validator = new Validator();
   $validator->addRule('email', 'email');
   $validator->addRule('role', 'in:admin,user');
   $this->setValidator($validator);
   

4. Event Handling: Leverage Symfony events (e.g., data_import.pre_process, data_import.post_process) via the bundle:

   # config/services.yaml
   services:
       App\EventListener\ImportListener:
           tags:
               - { name: kernel.event_listener, event: data_import.pre_process, method: onPreProcess }
   

5. Chunked Processing: For large datasets, use ChunkReader or DatabaseWriter with batch_size:

   $reader = new ChunkReader(new CsvReader('large_file.csv'), 1000);
   

Integration Tips

• Doctrine ORM: Works seamlessly with Doctrine entities. Ensure entities have proper setters/getters.
• Symfony Forms: Use for file uploads:

  $form = $this->createFormBuilder()
      ->add('file', FileType::class)
      ->getForm();
  

• Logging: Wrap imports in a service with logging:

  $this->logger->info('Starting import for ' . $entityClass);
  $import->run();
  $this->logger->info('Import completed with ' . $import->getTotalRows() . ' rows');
  

Gotchas and Tips

Pitfalls

1. Memory Limits: Large CSV files may hit memory limits. Use ChunkReader or stream processing:

   $reader = new ChunkReader(new CsvReader('file.csv'), 500);
   

2. Doctrine Flush: DatabaseWriter flushes the entity manager after each row by default. For bulk inserts, disable auto-flush and manually flush:

   $writer = new DatabaseWriter(User::class, ['flush_mode' => DatabaseWriter::FLUSH_MANUAL]);
   $import->run();
   $entityManager->flush(); // Call once after all rows
   

3. Field Mismatches: If getFields() keys don’t match CSV headers or entity properties, imports will silently fail. Validate with:

   $this->getReader()->getHeaderRow(); // Check CSV headers
   

4. Symfony Cache: The bundle may cache configurations. Clear cache after updates:

   php bin/console cache:clear
   

Debugging

• Enable Verbose Output: Set the verbose option in the bundle config (config/packages/ddeboer_data_import.yaml):

  ddeboer_data_import:
      verbose: true
  

• Check Events: Use Symfony’s debug toolbar to inspect dispatched events (e.g., data_import.row_process).

Tips

1. Custom Writers: Extend AbstractWriter for non-Doctrine destinations (e.g., Elasticsearch):

   class ElasticsearchWriter extends AbstractWriter
   {
       public function write(array $row): void
       {
           $this->client->index(['body' => $row]);
       }
   }
   

2. Dry Runs: Use dry_run: true to validate without persisting:

   $writer = new DatabaseWriter(User::class, ['dry_run' => true]);
   

3. Error Handling: Implement RowProcessorInterface to handle errors per row:

   public function processRow(array $row): ?array
   {
       if (empty($row['email'])) {
           $this->logger->error('Skipping row: missing email');
           return null; // Skip this row
       }
       return $row;
   }
   

4. Performance:

   • Disable Doctrine proxies for bulk imports:

     $entityManager->getConfiguration()->setProxyDir(null);
     $entityManager->getConfiguration()->setProxyNamespace(null);
     

   • Use INSERT ... ON DUPLICATE KEY UPDATE for MySQL via a custom writer.

5. Testing: Mock readers/writers for unit tests:

   $mockReader = $this->createMock(CsvReader::class);
   $mockReader->method('read')->willReturn([['data' => 'row1']]);
   $import->setReader($mockReader);