Export Engine Laravel Package

Getting Started

Minimal Setup

1. Installation

   composer require turbostream/export-engine
   

   Publish the config file:

   php artisan vendor:publish --provider="TurboStream\ExportEngine\ExportEngineServiceProvider"
   

2. Basic Export (CSV Example)

   use TurboStream\ExportEngine\Facades\ExportEngine;
   
   $export = ExportEngine::query(YourModel::class)
       ->select(['id', 'name', 'email'])
       ->export('csv');
   

3. First Use Case: Quick CSV Export

   $export = ExportEngine::query(User::class)
       ->select(['id', 'name', 'created_at'])
       ->export('csv', 'users_export_' . now()->format('Y-m-d'));
   

   • Automatically downloads the file with timestamped filename.
   • No manual file handling required.

Implementation Patterns

Core Workflows

1. Chunked Query Processing

// Process 15K records per chunk (adjustable via config)
$export = ExportEngine::query(Product::class)
    ->chunk(15000)
    ->select(['sku', 'price', 'stock'])
    ->export('xlsx');

• Best Practice: Use cursor() for memory efficiency (default behavior).
• Configurable: Adjust chunk_size in config/export-engine.php.

2. PDF with Advanced Features

$export = ExportEngine::query(Order::class)
    ->select(['order_id', 'customer', 'total'])
    ->pdf()
    ->setTitle('Yearly Orders Report')
    ->addSubtotal('total', 'SUM', 'quarter')
    ->addGrandTotal('total', 'SUM')
    ->export();

• Key Methods:
  • setTitle(): PDF title/header.
  • addSubtotal(): Group subtotals (e.g., by quarter).
  • addGrandTotal(): Summarize entire dataset.
  • setColumnWidths(): Customize PDF column widths.

3. Async Export with Queues

// Dispatch to queue (uses Redis by default)
$export = ExportEngine::query(Transaction::class)
    ->select(['date', 'amount', 'status'])
    ->export('pdf', 'transactions_report', true); // 3rd param = async

• Queue Job: TurboStream\ExportEngine\Jobs\ExportJob.
• Check Status:

  $status = ExportEngine::status('transactions_report');
  if ($status->completed) {
      return redirect($status->downloadUrl);
  }
  

4. Customizing Export Headers

$export = ExportEngine::query(Invoice::class)
    ->select(['invoice_number', 'client', 'amount'])
    ->setHeaders([
        'invoice_number' => 'Invoice #',
        'client' => 'Client Name',
        'amount' => 'Amount ($)',
    ])
    ->export('docx');

5. SQL Export for Backups

$export = ExportEngine::query(Post::class)
    ->select('*')
    ->sql()
    ->setFileName('posts_backup_' . now()->format('Y-m-d'))
    ->export();

• Generates a .sql file with INSERT statements.

Integration Tips

Laravel Routes/Controller

use TurboStream\ExportEngine\Facades\ExportEngine;

Route::get('/export-users', function () {
    return ExportEngine::query(User::class)
        ->select(['id', 'name', 'email'])
        ->export('csv');
});

Blade Integration (for Async Download Links)

@if(ExportEngine::status($exportId)->pending)
    <a href="{{ route('export.status', $exportId) }}">Check Status</a>
@elseif(ExportEngine::status($exportId)->completed)
    <a href="{{ ExportEngine::status($exportId)->downloadUrl }}">Download</a>
@endif

Testing Exports

public function testExport()
{
    $export = ExportEngine::query(User::class)
        ->select(['id', 'name'])
        ->export('csv');

    // Assert file was generated (check storage/temp)
    $this->assertFileExists(storage_path('app/temp/users_export_'.now()->format('Y-m-d').'.csv'));
}

Gotchas and Tips

Pitfalls

1. Memory Leaks with Large Datasets

   • Issue: Forgetting to use cursor() or setting an overly large chunk_size.
   • Fix: Always use cursor() for datasets >10K records. Configure chunk_size in config/export-engine.php (default: 10,000).

     'chunk_size' => env('EXPORT_CHUNK_SIZE', 15000),
     

2. PDF Generation Timeouts

   • Issue: Generating 9000+ page PDFs may hit PHP’s max_execution_time.
   • Fix: Use async export (export('pdf', 'filename', true)) or increase max_execution_time in php.ini temporarily.
   • Workaround: Split PDFs into multiple files by date ranges.

3. Queue Failures

   • Issue: Redis queue worker not running or misconfigured.
   • Fix: Ensure queue:work is running:

     php artisan queue:work --daemon
     

   • Debug: Check failed_jobs table for errors:

     php artisan queue:failed-table
     

4. Column Width Issues in PDF

   • Issue: Text overflow in PDF cells.
   • Fix: Use setColumnWidths() or adjust font size:

     ->pdf()
     ->setColumnWidths([10, 50, 30]) // widths in mm
     ->setFontSize(8)
     

5. SQL Export Quirks

   • Issue: Special characters in data breaking SQL syntax.
   • Fix: Escape data or use setSqlEscape(true):

     ->sql()
     ->setSqlEscape(true)
     

Debugging Tips

1. Log Export Progress Enable debug logs in config/export-engine.php:

   'debug' => env('EXPORT_DEBUG', false),
   

   • Logs chunk processing to storage/logs/export-engine.log.

2. Check Temp Files

   • Exports are stored in storage/app/temp/ before download.
   • Verify files exist here if downloads fail.

3. Validate Query Test your query manually first:

   YourModel::cursor()->take(10)->get(); // Check for errors
   

4. Async Export Status

   $status = ExportEngine::status('export_id');
   // $status->progress (0-100), $status->completed, $status->error
   

Extension Points

1. Custom Export Formats Extend the TurboStream\ExportEngine\Contracts\ExportFormat interface:

   class CustomFormat implements ExportFormat {
       public function generate(array $data, string $fileName) { ... }
   }
   

   Register in config/export-engine.php:

   'formats' => [
       'custom' => \App\Exports\CustomFormat::class,
   ],
   

2. Override Default Styling Publish and modify the PDF/DOCX templates:

   php artisan vendor:publish --tag=export-engine-views
   

   • Templates located in resources/views/vendor/export-engine/.

3. Hooks for Pre/Post Processing Use events:

   // In EventServiceProvider
   ExportEngine::exporting(function ($export) {
       // Modify data before export
   });
   
   ExportEngine::exported(function ($export) {
       // Cleanup after export
   });
   

4. Custom Chunk Processing Override chunk logic for complex data:

   $export = ExportEngine::query(ComplexModel::class)
       ->setChunkProcessor(function ($chunk) {
           return $chunk->map(fn ($item) => [
               'id' => $item->id,
               'processed_data' => $this->process($item),
           ]);
       })
       ->export('csv');
   

Performance Optimizations

1. Database Indexes Ensure select() columns are indexed for faster queries.

2. Queue Batch Size Adjust queue_batch_size in config to balance memory/CPU:

   'queue_batch_size' => 5, // Process 5 jobs at once
   

3. Storage Engine For async exports, use a fast storage driver (e.g., local or s3):

   'storage_disk' => 's3',
   

4. Disable Logging in Production

   'debug'