Laravel Data Jobs Laravel Package

Getting Started

1. Installation:

   composer require badrshs/laravel-data-jobs
   php artisan data-jobs:install
   

   This creates the data_jobs_log table and publishes the config file.

2. First Use Case: Convert a one-time data migration command into a trackable job:

   php artisan make:command MigrateLegacyUsers
   

   Add the DataJobable trait and define priority:

   use Badrshs\LaravelDataJobs\Contracts\DataJobable;
   
   class MigrateLegacyUsers extends Command {
       use DataJobable;
   
       public function getJobPriority(): int { return 5; } // Lower = higher priority
   }
   

3. Run Jobs:

   php artisan data:run-jobs
   

   The package auto-discovers all DataJobable commands, sorts them by priority, and executes pending jobs.

Implementation Patterns

1. Job Definition Workflow

• Basic Job:

  class BackfillUserProfiles extends Command {
      use DataJobable;
  
      public function handle() {
          // Migration logic
          return self::SUCCESS;
      }
  }
  

• Priority & Metadata:

  public function getJobPriority(): int { return 1; } // Critical job
  public function getJobParameters(): array { return ['batch_size' => 1000]; }
  

2. Integration with Laravel Queues

• Async Execution:

  // In your job command:
  public function handle() {
      dispatch(new ProcessLargeDataset)->delay(now()->addMinutes(5));
  }
  

• Queue Listener: Add a queue worker to process jobs in the background:

  php artisan queue:work --queue=data-jobs
  

3. Conditional Job Execution

• Dynamic Enablement:

  public function isEnabled(): bool {
      return config('app.env') === 'production';
  }
  

• Environment-Specific Jobs: Use config() or environment variables to toggle jobs.

4. Job Chaining

• Dependent Jobs: Use getJobPriority() to enforce order (e.g., schema_migration → data_backfill).

  // Run schema migration first (priority 1)
  public function getJobPriority(): int { return 1; }
  

5. Testing Jobs

• Unit Test:

  public function test_job_execution() {
      $job = new MigrateLegacyUsers();
      $this->assertEquals(1, $job->getJobPriority());
      $this->assertTrue($job->isEnabled());
  }
  

• Integration Test: Mock the data_jobs_log table and verify status transitions:

  $this->artisan('data:run-jobs')
       ->expectsOutput('✅ Completed: MigrateLegacyUsers');
  

Gotchas and Tips

Pitfalls

1. Priority Collisions:

   • Jobs with the same priority may run in an undefined order. Use unique priorities (e.g., 1, 2, 3 instead of 1, 1, 2).

2. Database Locks:

   • Long-running jobs can block the data_jobs_log table. Use transactions or queue jobs for heavy operations:

     DB::transaction(function () {
         // Job logic
     });
     

3. Idempotency:

   • Ensure jobs are idempotent (safe to re-run). Use getJobParameters() to track state:

     public function handle() {
         if ($this->alreadyProcessed()) {
             return self::SUCCESS;
         }
         // Process data
     }
     

4. Configuration Overrides:

   • Disabling logging ('logging_enabled' => false) bypasses tracking. Use sparingly:

     // config/data-jobs.php
     'logging_enabled' => env('DATA_JOBS_LOGGING', true),
     

5. Artisan Command Discovery:

   • Jobs must be registered as Artisan commands (e.g., via Console/Kernel.php). Unregistered jobs are skipped.

Debugging Tips

• Check Logs:

  php artisan data:run-jobs --verbose
  

  Outputs detailed job statuses (e.g., ▶️ Running: JobName).

• Inspect Database:

  SELECT * FROM data_jobs_log ORDER BY created_at DESC;
  

  Verify job statuses (pending, running, completed, failed).

• Force Re-Run:

  php artisan data:run-jobs --force
  

  Useful for testing or recovering from failures.

Extension Points

1. Custom Statuses: Extend the data_jobs_log table to add custom fields (e.g., started_at, ended_at):

   Schema::table('data_jobs_log', function (Blueprint $table) {
       $table->timestamp('started_at')->nullable();
   });
   

2. Event Listeners: Trigger events for job lifecycle hooks (e.g., JobStarting, JobFailed):

   // In DataJobServiceProvider
   Event::listen(JobStarting::class, function ($job) {
       Log::info("Job {$job->getName()} starting...");
   });
   

3. Slack Notifications: Integrate with Laravel Notifications:

   public function handle() {
       try {
           // Job logic
       } catch (\Exception $e) {
           Notification::route('slack', config('services.slack.webhook'))
               ->notify(new JobFailed($this->getName(), $e));
       }
   }
   

4. Parallel Execution: Use Laravel Queues to run jobs concurrently (bypass priority order):

   // In DataJobServiceProvider
   public function boot() {
       DataJob::runJobs()->each(function ($job) {
           dispatch($job)->onQueue('data-jobs');
       });
   }
   

Performance Quirks

• Large Job Batches: For jobs processing >10K records, chunk data to avoid timeouts:

  public function handle() {
      User::chunk(1000, function ($users) {
          // Process chunk
      });
  }
  

• Queue Workers: Scale workers based on job volume:

  php artisan queue:work --queue=data-jobs --sleep=3 --tries=3
  

Migration Safety

• Backup Data: Always back up critical tables before running destructive jobs:

  mysqldump -u user -p db_name table_name > backup.sql
  

• Dry Runs: Add a --dry-run flag to simulate jobs without changes:

  protected $signature = 'data:migrate-users {--dry-run}';
  public function handle() {
      if ($this->option('dry-run')) {
          $this->info("Dry run: Would migrate {$this->countUsers()} users.");
          return self::SUCCESS;
      }
      // Actual migration
  }