Mathielen Import Engine Laravel Package

Getting Started

Minimal Setup

1. Installation

   composer require mathielen/import-engine
   

   Ensure ddeboer/data-import is also installed (dependency).

2. First Use Case: CSV Import

   • Define a mapping between CSV columns and your Eloquent model fields.
   • Example: Import users.csv into a User model:

     use Mathielen\ImportEngine\ImportEngine;
     use Ddeboer\DataImport\Reader\CsvReader;
     
     $reader = new CsvReader('path/to/users.csv');
     $importEngine = new ImportEngine(new UserImportMapping());
     $results = $importEngine->import($reader);
     

   • Key Files to Review:
     • src/ImportEngine.php (core logic).
     • src/Mapping/AbstractMapping.php (base mapping class).
     • tests/ (real-world examples).

Implementation Patterns

Core Workflow

1. Define a Mapping Class Extend Mathielen\ImportEngine\Mapping\AbstractMapping and override:

   • getModel() → Return your Eloquent model class.
   • getColumns() → Define CSV/array keys → model attributes.
   • getRules() → Validation rules (e.g., ['email' => 'required|email']).
   • Example:

     class UserImportMapping extends AbstractMapping {
         public function getModel() { return User::class; }
         public function getColumns() {
             return [
                 'email' => 'email',
                 'name' => 'name',
                 'role' => 'role_id',
             ];
         }
         public function getRules() {
             return ['email' => 'required|unique:users'];
         }
     }
     

2. Handle Different Data Sources

   • CSV: Use CsvReader (default).
   • Excel: Use League\Csv\ExcelReader (composer require league/csv).
   • API/JSON: Parse into an array and use ArrayReader.

     $reader = new ArrayReader(json_decode(file_get_contents('data.json'), true));
     

3. Batch Processing Use ChunkReader to process large files in chunks:

   $reader = new ChunkReader(new CsvReader('large_file.csv'), 100);
   $importEngine->import($reader);
   

4. Post-Import Actions

   • Listen for events (ImportStarted, RowImported, ImportCompleted):

     $importEngine->on('row.imported', function ($row) {
         // Log or trigger side-effects (e.g., send welcome email).
     });
     

5. Exporting Data Reuse mappings for exports:

   $exporter = new ImportEngine(new UserImportMapping());
   $exporter->export(User::all())->toCsv('users_export.csv');
   

Integration Tips

• Laravel Service Provider Bind the engine and mappings in AppServiceProvider:

  $this->app->singleton(ImportEngine::class, function ($app) {
      return new ImportEngine(new UserImportMapping());
  });
  

• Artisan Command Create a CLI command for imports:

  class ImportUsersCommand extends Command {
      protected $signature = 'import:users {file}';
      public function handle() {
          $engine = app(ImportEngine::class);
          $results = $engine->import(new CsvReader($this->argument('file')));
          $this->info("Imported {$results['imported']} users.");
      }
  }
  

• Queue Jobs For long-running imports, dispatch a job:

  dispatch(new ImportUsersJob('path/to/file.csv'));
  

  class ImportUsersJob implements ShouldQueue {
      public function handle() {
          $engine = app(ImportEngine::class);
          $engine->import(new CsvReader($this->filePath));
      }
  }
  

Gotchas and Tips

Pitfalls

1. Column Mismatches

   • Issue: CSV headers don’t match getColumns() keys.
   • Fix: Use setHeaderRow() on the reader or override mapRow() in your mapping:

     public function mapRow(array $row) {
         return [
             'email' => $row['Email'], // Case-sensitive!
             'name' => $row['Full Name'],
         ];
     }
     

2. Validation Errors

   • Issue: Silent failures if validation rules aren’t strict.
   • Fix: Enable detailed error logging:

     $importEngine->setLogger(new SingleLineLogger());
     $results = $importEngine->import($reader);
     dd($results['errors']); // Inspect failed rows.
     

3. Memory Limits

   • Issue: Large files crash with Allowed memory exhausted.
   • Fix: Use ChunkReader or increase memory_limit in php.ini.

4. Model Relationships

   • Issue: Nested imports (e.g., User with Address) require manual handling.
   • Fix: Use afterImport to resolve relationships:

     public function afterImport() {
         User::with('address')->find($this->getImportedIds())->get();
     }
     

Debugging

• Enable Verbose Logging

  $importEngine->setLogger(new StreamHandler(storage_path('logs/import.log'), Logger::DEBUG));
  

• Test with Small Files Start with a 5-row CSV to validate mappings before scaling.

Extension Points

1. Custom Readers Implement Ddeboer\DataImport\Reader\ReaderInterface for unsupported formats (e.g., XML):

   class XmlReader implements ReaderInterface {
       public function getRecords() { /* ... */ }
   }
   

2. Pre/Post-Processing Override beforeImport() or afterImport() in your mapping:

   public function beforeImport() {
       DB::beginTransaction();
   }
   public function afterImport() {
       DB::commit();
   }
   

3. Custom Error Handling Extend Mathielen\ImportEngine\ImportEngine to modify error responses:

   class CustomImportEngine extends ImportEngine {
       protected function handleRowErrors(array $errors) {
           // Custom logic (e.g., notify admin).
       }
   }
   

4. Database Transactions Wrap imports in transactions for data integrity:

   DB::transaction(function () use ($engine, $reader) {
       $engine->import($reader);
   });
   

Config Quirks

• Time Zone Handling Ensure config/app.php timezone matches your import data to avoid timestamp issues.
• Locale-Specific Data Use setLocale() on the reader for date/number parsing:

  $reader->setLocale('de_DE');