Data Import Laravel Package

Getting Started

First Steps

1. Installation Add the package via Composer:

   composer require ddeboer/data-import
   

   Publish the config file (if needed):

   php artisan vendor:publish --provider="Ddeboer\DataImport\DataImportServiceProvider"
   

2. Basic Usage Import a CSV file:

   use Ddeboer\DataImport\Reader\CsvReader;
   use Ddeboer\DataImport\Reader\Header\HeaderStyle;
   
   $reader = new CsvReader('path/to/file.csv', 'r');
   $reader->setHeaderStyle(HeaderStyle::NAMED);
   $reader->setHeaderRow(1); // Skip header row if needed
   
   foreach ($reader as $record) {
       // Process each record (array of data)
   }
   

3. First Export Export data to CSV:

   use Ddeboer\DataImport\Writer\CsvWriter;
   
   $writer = new CsvWriter('path/to/output.csv', 'w');
   $writer->addRow(['Name', 'Email']); // Header
   $writer->addRow(['John Doe', 'john@example.com']);
   $writer->close();
   

Implementation Patterns

Common Workflows

1. Bulk Data Import Use BatchReader for large files to avoid memory issues:

   $reader = new BatchReader(new CsvReader('large_file.csv'), 1000);
   foreach ($reader as $batch) {
       // Process batch (array of records)
   }
   

2. Mapping Fields to Eloquent Models Combine with Laravel’s Model::create() or Model::updateOrCreate():

   foreach ($reader as $record) {
       User::updateOrCreate(
           ['email' => $record['email']],
           ['name' => $record['name']]
       );
   }
   

3. Validation During Import Use Laravel’s Validator inside the loop:

   $validator = Validator::make($record, [
       'email' => 'required|email',
       'name' => 'required|string',
   ]);
   if ($validator->fails()) {
       // Log errors or skip invalid records
   }
   

4. Exporting Eloquent Collections Convert a collection to CSV:

   $users = User::all();
   $writer = new CsvWriter('users.csv', 'w');
   $writer->addRow(['Name', 'Email']); // Header
   $users->each(fn ($user) => $writer->addRow([
       $user->name,
       $user->email,
   ]));
   $writer->close();
   

5. Handling Different Formats

   • Excel (XLSX): Use ExcelReader/ExcelWriter (requires phpoffice/phpexcel).
   • JSON: Use JsonReader/JsonWriter for structured data.
   • XML: Use XmlReader/XmlWriter for hierarchical data.

6. Chunking for Large Exports Use StreamWriter to avoid memory overload:

   $writer = new StreamWriter(new CsvWriter('large_export.csv', 'w'));
   foreach ($users->chunk(1000) as $chunk) {
       $chunk->each(fn ($user) => $writer->addRow([$user->name, $user->email]));
   }
   $writer->close();
   

Gotchas and Tips

Pitfalls

1. Memory Issues

   • Problem: Large files can exhaust memory with CsvReader.
   • Fix: Use BatchReader or StreamWriter for chunked processing.
   • Tip: Monitor memory with memory_get_usage() in loops.

2. Encoding Problems

   • Problem: CSV files with non-UTF-8 encoding may corrupt data.
   • Fix: Specify encoding in CsvReader:

     $reader = new CsvReader('file.csv', 'r');
     $reader->setEncoding('ISO-8859-1');
     

3. Header Mismatches

   • Problem: Column names in CSV may not match your expected keys.
   • Fix: Use HeaderStyle::FIRST_ROW and manually map fields:

     $reader->setHeaderStyle(HeaderStyle::FIRST_ROW);
     $mappedRecord = [
         'name' => $record['Name'], // Case-sensitive!
         'email' => $record['Email'],
     ];
     

4. File Locking

   • Problem: Writing to files while they’re open can cause errors.
   • Fix: Always call $writer->close() after finishing.

5. Deprecated Methods

   • Problem: Some methods (e.g., setDelimiter()) may behave unexpectedly in newer PHP versions.
   • Fix: Check the docs for updates or use alternatives like setEnclosure().

Debugging Tips

• Log Invalid Records:

  $invalidRecords = [];
  foreach ($reader as $record) {
      if (!$validator->passes()) {
          $invalidRecords[] = $record;
          continue;
      }
  }
  file_put_contents('invalid_records.log', json_encode($invalidRecords));
  

• Test with Small Files First: Validate logic on a 10-row CSV before scaling.
• Use getHeaders(): Inspect column names dynamically:

  print_r($reader->getHeaders());
  

Extension Points

1. Custom Readers/Writers Extend AbstractReader or AbstractWriter for proprietary formats:

   class JsonApiReader extends AbstractReader {
       public function read() { /* Custom logic */ }
   }
   

2. Event Hooks Trigger events during import/export (e.g., import.start, record.processed):

   event(new DataImportStarted($reader));
   

3. Laravel Service Provider Bind readers/writers to the container for dependency injection:

   $this->app->bind(CsvReader::class, function () {
       return new CsvReader('file.csv');
   });
   

4. Queue Jobs for Large Imports Offload processing to queues:

   foreach ($reader as $record) {
       ImportRecordJob::dispatch($record);
   }