Chunker Laravel Package

Getting Started

Minimal Steps

1. Installation:

   composer require jstewmc/chunker
   

   Ensure ext-mbstring is enabled in your PHP environment.

2. First Use Case: Process a large UTF-8 encoded file without breaking multi-byte characters:

   use Jstewmc\Chunker\File;
   
   $chunker = new File('large-file.csv', 'UTF-8', 8192); // 8KB chunks
   while (false !== ($chunk = $chunker->current())) {
       process($chunk); // Your logic here
       $chunker->next();
   }
   

3. Where to Look First:

   • README.md: Focus on the "Consuming the chunks" section for core usage.
   • Changelog: Review 0.2.1 for fixes related to malformed byte sequences and idempotency.

Implementation Patterns

Core Workflows

1. File Processing:

   $chunker = new File('data/export.jsonl', 'UTF-8', 4096);
   while ($chunker->hasNextChunk()) {
       $data = json_decode($chunker->current(), true);
       $chunker->next();
       // Process line-by-line JSON data
   }
   

2. String Chunking:

   $chunker = new Text($largeString, 'UTF-8', 2000); // 2000 chars
   foreach ($chunker as $chunk) {
       // Process text in 2000-character blocks
   }
   

3. Memory-Efficient CSV Parsing:

   $chunker = new File('huge.csv', 'UTF-8', 1024);
   $headers = null;
   while (false !== ($chunk = $chunker->current())) {
       if (!$headers) {
           $headers = str_getcsv($chunk);
           $chunker->next();
           continue;
       }
       $row = str_getcsv($chunk);
       // Process row with $headers
       $chunker->next();
   }
   

Integration Tips

• Laravel Filesystem: Combine with Laravel's Storage facade for cloud storage:

  use Illuminate\Support\Facades\Storage;
  use Jstewmc\Chunker\File;
  
  $path = Storage::path('backups/large-backup.sql');
  $chunker = new File($path, 'UTF-8', 16384); // 16KB chunks
  

• Queue Jobs: Process chunks in background jobs:

  $chunker = new File('data/process', 'UTF-8', 8192);
  while ($chunker->hasNextChunk()) {
      ProcessChunkJob::dispatch($chunker->current())
          ->afterRelease(fn() => $chunker->next());
  }
  

• Streaming Responses:

  $chunker = new File('reports/generated.pdf', null, 8192);
  return response()->stream(fn() => yield $chunker->current(), $chunker->next());
  

Gotchas and Tips

Pitfalls

1. Encoding Assumptions:

   • Gotcha: Omitting the encoding defaults to mb_internal_encoding(), which may cause issues if your app uses UTF-8 but the file uses ISO-8859-1.
   • Fix: Always specify encoding explicitly:

     $chunker = new File('file.txt', 'ISO-8859-1'); // Force correct encoding
     

2. Chunk Size Limits:

   • Gotcha: Setting size=0 or negative values throws InvalidArgumentException.
   • Fix: Validate sizes:

     $size = max(1, (int)$request->input('chunk_size', 8192));
     

3. Navigation Edge Cases:

   • Gotcha: next()/previous() are idempotent but not reversible (forward/backward chunks may differ).
   • Fix: Reset with reset() before bidirectional traversal:

     $chunker->next(); // Move forward
     $chunker->reset(); // Reset to start
     

4. Malformed Bytes:

   • Gotcha: If the input file/string has corrupted UTF-8, chunks may still contain ?.
   • Fix: Pre-validate input or use mb_convert_encoding():

     $cleanText = mb_convert_encoding($dirtyText, 'UTF-8', 'UTF-8//IGNORE');
     

Debugging Tips

1. Verify Chunk Counts:

   $chunker = new Text($largeString, 'UTF-8', 100);
   dd($chunker->countChunks()); // Debug total chunks
   

2. Inspect Chunk Boundaries:

   $chunker = new File('file.txt', 'UTF-8');
   while ($chunker->hasNextChunk()) {
       $chunk = $chunker->current();
       $chunkLength = mb_strlen($chunk, 'UTF-8');
       dd($chunk, $chunkLength); // Check for unexpected lengths
       $chunker->next();
   }
   

3. Performance Profiling:

   • Use memory_get_usage() to compare chunk sizes:

     $chunker = new File('large.log', 'UTF-8', 8192);
     while ($chunker->hasNextChunk()) {
         $start = memory_get_usage();
         $chunk = $chunker->current();
         $usage = memory_get_usage() - $start;
         dd($usage); // Monitor memory per chunk
         $chunker->next();
     }
     

Extension Points

1. Custom Chunk Validators:

   $chunker = new File('data.json', 'UTF-8');
   while ($chunker->hasNextChunk()) {
       $chunk = $chunker->current();
       if (!json_validate($chunk)) { // Custom validation
           throw new \RuntimeException("Invalid JSON chunk");
       }
       $chunker->next();
   }
   

2. Lazy-Loading with Generators:

   function chunkGenerator(File $chunker) {
       while ($chunker->hasNextChunk()) {
           yield $chunker->current();
           $chunker->next();
       }
   }
   foreach (chunkGenerator($chunker) as $chunk) {
       // Process lazily
   }
   

3. Event-Based Processing:

   $chunker = new File('stream.log', 'UTF-8');
   $chunker->on('chunk', fn($chunk) => log($chunk));
   $chunker->on('end', fn() => notify('Processing complete'));
   while ($chunker->hasNextChunk()) {
       $chunker->next();
   }
   

   (Note: Requires extending the class to support events.)

4. Progress Tracking:

   $totalChunks = $chunker->countChunks();
   $processed = 0;
   while ($chunker->hasNextChunk()) {
       $chunk = $chunker->current();
       $progress = ($processed++ / $totalChunks) * 100;
       reportProgress($progress);
       $chunker->next();
   }
   

Config Quirks

• Default Sizes:
  • Files: 8,192 bytes (adjust for I/O performance).
  • Strings: 2,000 characters (max ~8,000 bytes for UTF-8).
• Windows Line Endings:
  • Use mb_convert_encoding() to normalize before chunking:

    $normalized = mb_convert_encoding($text, 'UTF-8', 'UTF-8');
    $chunker = new Text($normalized, 'UTF-8');