Job Queue Bundle Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require daanbiesterbos/job-queue-bundle
   

2. Register Bundle: Add to config/bundles.php:

   JMS\JobQueueBundle\JMSJobQueueBundle::class => ['all' => true],
   

3. Prepare Job Queue Console: Copy and modify bin/console to bin/job-queue:

   cp bin/console bin/job-queue
   

   Replace use Symfony\Bundle\FrameworkBundle\Console\Application; with:

   use JMS\JobQueueBundle\Console\Application;
   

4. First Job Creation: Use the Job entity to queue a command:

   use JMS\JobQueueBundle\Entity\Job;
   
   $job = new Job();
   $job->setCommand('app:your-command');
   $job->setSchedule(new \DateTime('+1 hour')); // Optional: Schedule for later
   $job->setData(['arg1' => 'value1']); // Optional: Command arguments
   
   $em->persist($job);
   $em->flush();
   

5. Run the Queue Worker:

   php bin/job-queue run
   

First Use Case: Background Command Execution

Queue a command (e.g., app:send-emails) to run asynchronously:

$job = new Job();
$job->setCommand('app:send-emails');
$job->setData(['user_id' => 123]);
$em->persist($job);
$em->flush();

Trigger the worker in a separate terminal:

php bin/job-queue run

Implementation Patterns

Workflow: Job Creation and Execution

1. Job Creation:

   • Use Job entity to encapsulate command metadata (name, arguments, schedule).
   • Example: Queue a Laravel Artisan command (if using Symfony/Laravel hybrid):

     $job = new Job();
     $job->setCommand('cache:clear');
     $job->setSchedule(new \DateTime('now')); // Run immediately
     $em->persist($job);
     

2. Scheduling Jobs:

   • Schedule jobs for future execution:

     $job->setSchedule(new \DateTime('tomorrow'));
     

   • Use cron-like syntax for recurring jobs (via JobQueueBundle's CronJob entity if extended).

3. Job Data Serialization:

   • Pass complex data as JSON-serializable arrays:

     $job->setData(json_encode(['users' => [1, 2, 3], 'template' => 'welcome']));
     

4. Worker Management:

   • Run workers in a loop (default) or single-run mode:

     php bin/job-queue run --loop  # Default (continuous)
     php bin/job-queue run --once  # Single execution
     

   • Limit concurrency with --concurrency=N (e.g., --concurrency=5).

Integration Tips

1. Symfony/Laravel Hybrid:

   • For Laravel, use Symfony's Process component to call bin/job-queue:

     $process = new Process(['php', 'bin/job-queue', 'run', '--once']);
     $process->run();
     

   • Alternatively, create a Laravel Artisan command to wrap the bundle.

2. Database-Driven Jobs:

   • Extend the Job entity to add custom fields (e.g., priority, retries):

     /**
      * @ORM\Column(type="integer")
      */
     private $priority = 0;
     

   • Order jobs by priority in the worker:

     $jobs = $em->createQueryBuilder()
         ->select('j')
         ->from('JMSJobQueueBundle:Job', 'j')
         ->orderBy('j.priority', 'DESC')
         ->getQuery()
         ->getResult();
     

3. Event Listeners:

   • Listen for job events (e.g., JobCreated, JobExecuted) to log or notify:

     use JMS\JobQueueBundle\Event\JobEvent;
     
     $dispatcher->addListener(JobEvent::JOB_CREATED, function (JobEvent $event) {
         // Log job creation
     });
     

4. Testing:

   • Mock the Job repository in tests:

     $jobRepo = $this->createMock(JobRepository::class);
     $jobRepo->method('findNextJob')->willReturn($job);
     $this->container->set('jms_job_queue.job.repository', $jobRepo);
     

Gotchas and Tips

Pitfalls

1. Worker Stuck in Loop:

   • If the worker hangs, check for:
     • Unhandled exceptions in commands (log output with --verbose).
     • Locking issues (ensure Job entities are marked as executed after completion).
   • Solution: Add a retry mechanism or use --once to debug.

2. Command Arguments:

   • Non-serializable data (e.g., objects) in setData() will cause errors.
   • Fix: Convert data to arrays or JSON strings:

     $job->setData(['object' => json_encode($complexObject)]);
     

3. Database Locks:

   • Concurrent workers may race to claim jobs.
   • Solution: Use database transactions or add a locked_at column to Job:

     $job->setLockedAt(new \DateTime());
     $em->flush();
     

4. Timezone Issues:

   • Schedule uses the system timezone. Explicitly set timezone in jobs:

     $schedule = new \DateTime('+1 hour', new \DateTimeZone('UTC'));
     $job->setSchedule($schedule);
     

Debugging

1. Worker Logs:

   • Enable verbose output:

     php bin/job-queue run --verbose
     

   • Redirect logs to a file:

     php bin/job-queue run --verbose >> /var/log/job-queue.log 2>&1
     

2. Job Inspection:

   • Query the database directly to check job status:

     SELECT * FROM job WHERE executed_at IS NULL;
     

   • Use Symfony's profiler (if enabled) to track job events.

3. Command Execution:

   • Test commands manually first:

     php bin/console app:your-command --arg=value
     

   • Ensure commands are callable via bin/job-queue (same environment as bin/console).

Extension Points

1. Custom Job Types:

   • Extend the Job entity to support custom logic:

     namespace App\Entity;
     
     use JMS\JobQueueBundle\Entity\Job as BaseJob;
     
     class CustomJob extends BaseJob {
         private $customField;
     
         // Add getters/setters and ORM mappings
     }
     

2. Worker Hooks:

   • Override the worker class to add pre/post-execution logic:

     namespace App\JobQueue;
     
     use JMS\JobQueueBundle\Worker;
     
     class CustomWorker extends Worker {
         protected function executeJob(Job $job) {
             // Custom logic before execution
             parent::executeJob($job);
             // Custom logic after execution
         }
     }
     

   • Register the custom worker in services.yaml:

     JMS\JobQueueBundle\Worker: '@App\JobQueue\CustomWorker'
     

3. Queue Prioritization:

   • Implement a custom repository to sort jobs dynamically:

     namespace App\Repository;
     
     use Doctrine\ORM\EntityRepository;
     
     class JobRepository extends EntityRepository {
         public function findNextJob() {
             return $this->createQueryBuilder('j')
                 ->where('j.executed_at IS NULL')
                 ->orderBy('j.priority', 'DESC')
                 ->getQuery()
                 ->getOneOrNullResult();
         }
     }
     

   • Update services.yaml to use the custom repository:

     jms_job_queue.job.repository: '@App\Repository\JobRepository'
     

4. Retry Mechanism:

   • Add a retries column to Job and implement exponential backoff:

     if ($job->getRetries() >= 3) {
         throw new \RuntimeException('Max retries exceeded');
     }
     $job->setRetries($job->getRetries() + 1);
     $job->setNextRun(new \DateTime('+1 minute'));
     

Configuration Quirks

1. Environment Variables:

   • The bundle reads JMS_JOB_QUEUE_CONCURRENCY for default concurrency.
   • Override in .env:

     JMS_JOB_QUEUE_CONCURRENCY=3
     

2. Doctrine Configuration:

   • Ensure the Job entity is mapped