Symfony File Importer Bundle Laravel Package

Getting Started

Minimal Setup

1. Installation Add the bundle to your Symfony/Laravel (via Symfony Bridge) project:

   composer require cooolinho/symfony-file-importer-bundle
   

   Register the bundle in config/bundles.php (Symfony) or config/app.php (Laravel):

   return [
       // ...
       Cooolinho\FileImporterBundle\FileImporterBundle::class => ['all' => true],
   ];
   

2. Configuration Publish the default config:

   php artisan vendor:publish --tag=file-importer-config
   

   Update config/file_importer.php with your storage paths (e.g., local, s3).

3. First Use Case: Import a CSV Create a controller method to handle file uploads and imports:

   use Cooolinho\FileImporterBundle\Importer\CsvImporter;
   
   public function import(Request $request)
   {
       $file = $request->file('csv_file');
       $importer = new CsvImporter($file->getPathname());
       $data = $importer->import(); // Returns associative array of rows
       // Process $data (e.g., save to DB)
   }
   

Implementation Patterns

Core Workflows

1. File Handling

   • Uploads: Use Symfony’s UploadedFile or Laravel’s Illuminate\Http\UploadedFile to pass files to importers.
   • Storage: Configure supported drivers (local, s3, etc.) in config/file_importer.php:

     'storage' => [
         'default' => 'local',
         'disks' => [
             'local' => storage_path('app/imports'),
             's3' => [
                 'key' => 'your-aws-key',
                 'secret' => 'your-aws-secret',
                 'bucket' => 'your-bucket',
             ],
         ],
     ];
     

   • Validation: Extend Cooolino\FileImporterBundle\Validator\FileValidator to add custom rules (e.g., file size, MIME types).

2. Importer Classes

   • Built-in Importers:
     • CsvImporter: For CSV files (supports headers, delimiters).
     • ExcelImporter: For .xlsx/.xls (requires phpoffice/phpspreadsheet).
     • JsonImporter: For JSON files.
   • Usage:

     $importer = new CsvImporter($filePath, [
         'delimiter' => ';',
         'enclosure' => '"',
         'headers' => true,
     ]);
     $data = $importer->import();
     

3. Data Processing

   • Chunking: Process large files in chunks to avoid memory issues:

     $importer->setChunkSize(100); // Process 100 rows at a time
     while ($chunk = $importer->getNextChunk()) {
         // Save $chunk to DB
     }
     

   • Mapping: Transform imported data before saving:

     $data = array_map(function ($row) {
         return [
             'name' => $row['full_name'],
             'email' => strtolower($row['email']),
         ];
     }, $data);
     

4. Integration with Laravel

   • Forms: Use Laravel’s FormRequest for validation:

     use Illuminate\Foundation\Http\FormRequest;
     
     class ImportRequest extends FormRequest
     {
         public function rules()
         {
             return [
                 'csv_file' => 'required|file|mimes:csv,txt|max:10240',
             ];
         }
     }
     

   • Jobs: Offload imports to queues for long-running tasks:

     use Cooolinho\FileImporterBundle\Jobs\ImportFileJob;
     
     ImportFileJob::dispatch($filePath, CsvImporter::class)->onQueue('imports');
     

5. Event Listeners

   • Listen to import events (e.g., FileImportStarted, FileImportCompleted) via Symfony’s event dispatcher or Laravel’s event system:

     // In a service provider (Symfony)
     $dispatcher->addListener(FileImportEvent::IMPORT_STARTED, function (FileImportEvent $event) {
         Log::info('Import started for file: ' . $event->getFilePath());
     });
     

Gotchas and Tips

Pitfalls

1. Memory Limits

   • Issue: Large CSV/Excel files may exceed PHP’s memory_limit.
   • Fix: Use chunking ($importer->setChunkSize()) or increase memory_limit in php.ini.

2. Encoding Issues

   • Issue: CSV files with non-UTF-8 encoding (e.g., ISO-8859-1) may corrupt data.
   • Fix: Specify encoding in the importer:

     $importer = new CsvImporter($filePath, ['encoding' => 'ISO-8859-1']);
     

3. Excel Importer Dependencies

   • Issue: ExcelImporter requires phpoffice/phpspreadsheet, which isn’t auto-installed.
   • Fix: Install manually:

     composer require phpoffice/phpspreadsheet
     

4. File Paths in Laravel

   • Issue: Symfony’s file paths (e.g., /tmp/) may not work in Laravel’s storage system.
   • Fix: Use Laravel’s Storage facade to move files before importing:

     use Illuminate\Support\Facades\Storage;
     
     $filePath = Storage::disk('local')->putFile('imports', $request->file('csv_file'));
     

5. Case-Sensitive Headers

   • Issue: CSV headers may not match your expected keys (e.g., Name vs. name).
   • Fix: Normalize headers in the importer config:

     $importer = new CsvImporter($filePath, [
         'headers' => ['Name' => 'name', 'Email' => 'email'],
     ]);
     

Debugging Tips

1. Log Importer Output

   • Enable debug mode in config/file_importer.php:

     'debug' => true,
     

   • Check logs for raw data or errors.

2. Validate File Structure

   • Use getHeader() or getSample() methods to inspect file structure:

     $importer = new CsvImporter($filePath);
     $sample = $importer->getSample(5); // First 5 rows
     

3. Handle Malformed Data

   • Wrap imports in try-catch blocks:

     try {
         $data = $importer->import();
     } catch (\Cooolino\FileImporterBundle\Exception\ImportException $e) {
         Log::error('Import failed: ' . $e->getMessage());
         return back()->withError($e->getMessage());
     }
     

Extension Points

1. Custom Importers

   • Extend Cooolino\FileImporterBundle\Importer\AbstractImporter to support new file types (e.g., XML):

     class XmlImporter extends AbstractImporter
     {
         public function import()
         {
             $xml = simplexml_load_file($this->filePath);
             // Transform XML to array
             return json_decode(json_encode($xml), true);
         }
     }
     

2. Custom Validators

   • Extend FileValidator to add business logic:

     class CustomValidator extends FileValidator
     {
         public function validate($file)
         {
             if (!$this->isValidExtension($file, ['csv', 'txt'])) {
                 return false;
             }
             // Add custom checks (e.g., file hash)
             return true;
         }
     }
     

3. Event Customization

   • Dispatch custom events for your workflow:

     // In your importer service
     $dispatcher->dispatch(new FileImportEvent($filePath, 'custom.event'));
     

4. Storage Adapters

   • Add support for custom storage (e.g., Google Cloud) by implementing Cooolino\FileImporterBundle\Storage\StorageInterface.