Laravel Scout Bulk Actions Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require mozex/laravel-scout-bulk-actions
   

   Publish the config (optional):

   php artisan vendor:publish --provider="Mozex\ScoutBulkActions\ScoutBulkActionsServiceProvider" --tag="config"
   

2. First Use Case: Run a bulk import for all searchable models:

   php artisan scout:bulk-import
   

   For queued imports (recommended for large datasets):

   php artisan scout:bulk-queue-import
   

3. Where to Look First:

   • Config File: config/scout-bulk-actions.php (customize batch size, queue connection, or excluded models).
   • Commands: scout:bulk-import, scout:bulk-flush, scout:bulk-queue-import, scout:bulk-queue-flush.
   • Artisan Helpers: Check php artisan scout:bulk-import --help for options like --batch-size or --force.

Implementation Patterns

Core Workflows

1. Synchronous Bulk Import: Use for small datasets or development:

   php artisan scout:bulk-import --batch-size=100
   

   • Pattern: Ideal for CI/CD pipelines or local testing where speed isn’t critical.

2. Queued Bulk Import: For production or large datasets:

   php artisan scout:bulk-queue-import --batch-size=500 --queue=scout
   

   • Pattern: Dispatches jobs to a queue (e.g., scout connection) for background processing.
   • Monitoring: Use php artisan queue:work or a queue monitor (e.g., Laravel Horizon).

3. Selective Model Handling: Exclude specific models via config:

   // config/scout-bulk-actions.php
   'exclude' => [
       'App\Models\UnwantedModel',
   ],
   

   • Pattern: Useful for models with custom toSearchableArray() logic that shouldn’t run in bulk.

4. Progress Tracking: The package auto-generates a progress bar for synchronous imports. For queued jobs, log progress via job events:

   // In your job class (extends Mozex\ScoutBulkActions\Jobs\BulkImportJob)
   protected function handle(): void {
       $this->dispatchBrowserEvent('progress', ['model' => $this->model, 'progress' => $this->progress]);
   }
   

5. Integration with Scout Events: Listen to bulk actions via Scout’s events:

   Scout::flushing($modelClass, function () {
       // Pre-flush logic
   });
   

   • Pattern: Useful for analytics or cleanup before/after bulk operations.

Advanced Patterns

1. Custom Job Classes: Extend the base job to add pre/post-processing:

   namespace App\Jobs;
   
   use Mozex\ScoutBulkActions\Jobs\BulkImportJob;
   
   class CustomBulkImportJob extends BulkImportJob {
       public function handle() {
           // Custom logic before/after import
           parent::handle();
       }
   }
   

   • Config:

     'queue_job' => App\Jobs\CustomBulkImportJob::class,
     

2. Chunked Processing: Override getBatchSize() in a custom job to dynamically adjust batch sizes:

   public function getBatchSize(): int {
       return $this->model::count() < 1000 ? 200 : 500;
   }
   

3. Post-Import Actions: Chain commands or scripts after bulk imports:

   php artisan scout:bulk-import && php artisan optimize:clear
   

   • Pattern: Useful for triggering rebuilds of search-specific caches or analytics.

4. Testing: Mock bulk actions in tests:

   $this->artisan('scout:bulk-flush')
        ->expectsQuestion('Confirm flush', 'yes')
        ->assertExitCode(0);
   

Gotchas and Tips

Pitfalls

1. Queue Stuck Jobs:

   • Issue: Queued jobs may fail silently if the queue connection is misconfigured.
   • Fix: Monitor failed jobs with php artisan queue:failed-table and retry manually:

     php artisan queue:retry all
     

2. Memory Limits:

   • Issue: Large batch sizes (--batch-size) can hit PHP memory limits.
   • Fix: Start with batch-size=100 and increase incrementally. Use scout:bulk-queue-import for large datasets.

3. Model Auto-Discovery:

   • Issue: Models not being detected if they don’t use Searchable trait or have custom Scout configurations.
   • Fix: Explicitly list models in config:

     'models' => [
         App\Models\Product::class,
         App\Models\Article::class,
     ],
     

4. Race Conditions:

   • Issue: Concurrent bulk imports/flushes can corrupt indexes.
   • Fix: Use --force sparingly and ensure only one process runs at a time:

     php artisan scout:bulk-import --force
     

5. Scout Driver Compatibility:

   • Issue: Some Scout drivers (e.g., custom ones) may not work with bulk actions.
   • Fix: Test with your driver first. Check the driver compatibility list.

Debugging Tips

1. Verbose Output: Enable debug mode for detailed logs:

   php artisan scout:bulk-import --verbose
   

2. Job Logging: Configure the queue driver to log job failures (e.g., Redis with failed_table or database logging).

3. Progress Bar Issues:

   • Fix: Ensure your terminal supports ANSI escape codes. For CI/CD, use --no-progress:

     php artisan scout:bulk-import --no-progress
     

4. Custom Scout Logic:

   • Issue: Models with custom toSearchableArray() or shouldBeSearchable() may behave unexpectedly.
   • Fix: Test custom logic in isolation before bulk operations.

Extension Points

1. Custom Commands: Extend the base command to add project-specific logic:

   namespace App\Console\Commands;
   
   use Mozex\ScoutBulkActions\Console\BulkImportCommand;
   
   class CustomBulkImportCommand extends BulkImportCommand {
       protected $signature = 'scout:custom-import {--batch-size=100}';
       public function handle() {
           // Custom logic
           parent::handle();
       }
   }
   

2. Event Listeners: Listen to bulk action events for analytics or notifications:

   ScoutBulkActions::bulkImportStarted(function () {
       Log::info('Bulk import started');
   });
   

3. Dynamic Model Filtering: Override getModels() in a custom command to filter models dynamically:

   protected function getModels(): array {
       return Model::where('is_searchable', true)->get()->toArray();
   }
   

4. Post-Import Webhooks: Trigger HTTP requests or notifications after bulk imports:

   ScoutBulkActions::bulkImportCompleted(function () {
       Http::post('https://analytics.example.com/import-complete');
   });
   

Config Quirks

1. Batch Size:

   • Default: 100. Adjust based on your server’s memory (e.g., 500 for high-memory servers).
   • Tip: Smaller batches reduce memory usage but increase execution time.

2. Queue Connection:

   • Default: default. Specify a dedicated queue (e.g., scout) for bulk actions:

     'queue_connection' => 'scout',
     

3. Excluded Models:

   • Useful for models that shouldn’t participate in bulk actions (e.g., legacy or non-critical models).

4. Force Flag:

   • Warning: --force bypasses confirmation prompts. Use cautiously in production.

Performance Tips

1. Index Optimization:

   • Run scout:bulk-flush before bulk imports to clear stale data:

     php artisan scout:bulk-flush && php artisan scout:bulk-import
     

2. Database Indexes:

   • Ensure your Scout searchable column is indexed for faster bulk operations.

3. Queue Workers:

   • Scale queue workers (queue:work) based on bulk import size. Example for 10 workers:

     php artisan queue:work --queue=scout --daemon --tries=3 --sleep=3 --memory=512
     

4. Parallel Processing:

   • For multi-server setups, distribute bulk imports across servers using separate queue connections