Evil Queue Bundle Laravel Package

Getting Started

Minimal Setup

1. Installation

   composer require druidvav/evil-queue-bundle
   

   Add to config/bundles.php:

   return [
       // ...
       Druidvav\EvilQueueBundle\DruidvavEvilQueueBundle::class => ['all' => true],
   ];
   

2. Configure config/packages/dv_evil_queue.yaml

   dv_evil_queue:
       debug: '%kernel.debug%'
       connection: '@doctrine.dbal.xmlrpc_connection'  # Replace with your DBAL connection
       logger: '@monolog.logger.evil'                # Custom logger (optional)
       workers: 10                                   # Default worker count
       priority_workers: 5                           # Workers for high-priority jobs
   

3. First Use Case Create a job class:

   namespace App\Jobs;
   
   use Druidvav\EvilQueueBundle\Job\JobInterface;
   
   class SendEmailJob implements JobInterface
   {
       public function run(): void
       {
           // Job logic here
       }
   }
   

   Dispatch it in a controller/service:

   use Druidvav\EvilQueueBundle\Dispatcher\JobDispatcherInterface;
   
   class EmailController
   {
       public function __construct(private JobDispatcherInterface $dispatcher) {}
   
       public function sendWelcomeEmail(): void
       {
           $this->dispatcher->dispatch(new SendEmailJob());
       }
   }
   

4. Supervisor Setup Copy the template from docs/ to /etc/supervisor/conf.d/evil-queue.conf and adjust:

   [program:evil-queue]
   command=php /path/to/bin/console evil-queue:worker
   numprocs=10
   

5. Start Workers

   supervisorctl reread
   supervisorctl update
   supervisorctl start evil-queue:*
   

Implementation Patterns

Core Workflow

1. Job Design

   • Implement JobInterface for all jobs.
   • Use dependency injection for job dependencies (e.g., services, repositories).
   • Example with dependencies:

     class ProcessOrderJob implements JobInterface
     {
         public function __construct(
             private OrderRepository $orderRepo,
             private EmailService $emailService
         ) {}
     
         public function run(): void
         {
             $order = $this->orderRepo->find($this->orderId);
             $this->emailService->sendConfirmation($order);
         }
     }
     

2. Dispatching Jobs

   • Basic Dispatch:

     $dispatcher->dispatch(new SendEmailJob());
     

   • Delayed Jobs (if supported):

     $dispatcher->dispatch(new SendEmailJob(), 3600); // Delay in seconds
     

   • Priority Jobs:

     $dispatcher->dispatch(new CriticalJob(), 0, true); // High priority
     

3. Worker Management

   • Scaling: Adjust workers and priority_workers in config for load balancing.
   • Graceful Shutdown: Use evil-queue:stop-workers command to halt workers cleanly.
   • Monitoring: Check worker logs via monolog.logger.evil or Supervisor status:

     supervisorctl status
     

4. Integration with Symfony Events

   • Listen to job events (e.g., JobStarted, JobFailed) for logging/auditing:

     // config/services.yaml
     App\EventListener\JobListener:
         tags:
             - { name: kernel.event_listener, event: evil_queue.job_started, method: onJobStarted }
     

5. Database Schema

   • The bundle expects a table like evil_queue_jobs with columns:
     • id, payload, status, priority, created_at, processed_at.
   • Run migrations or use the provided schema SQL in docs/.

Advanced Patterns

1. Job Chaining Dispatch jobs sequentially in a worker:

   class ProcessOrderJob implements JobInterface
   {
       public function run(): void
       {
           $this->dispatcher->dispatch(new SendEmailJob());
           $this->dispatcher->dispatch(new UpdateInventoryJob());
       }
   }
   

   Note: Ensure the dispatcher is injected into the job.

2. Retry Logic Implement JobInterface::retry() for failed jobs:

   public function retry(): void
   {
       // Custom retry logic (e.g., exponential backoff)
   }
   

3. Bulk Processing Dispatch multiple jobs in a loop:

   foreach ($users as $user) {
       $this->dispatcher->dispatch(new SendNewsletterJob($user->id));
   }
   

4. Dynamic Worker Configuration Override worker count per environment:

   # config/packages/dev/dv_evil_queue.yaml
   dv_evil_queue:
       workers: 5  # Lower in dev
   

Gotchas and Tips

Pitfalls

1. Supervisor Misconfiguration

   • Issue: Workers crash silently if Supervisor isn’t properly configured.
   • Fix: Verify command points to the correct bin/console path and numprocs matches workers in config.
   • Debug: Check Supervisor logs:

     tail -f /var/log/supervisor/evil-queue-stderr.log
     

2. Database Locking

   • Issue: High concurrency may cause deadlocks on evil_queue_jobs.
   • Fix: Use priority_workers for critical jobs and monitor lock contention.

3. Job Serialization

   • Issue: Complex objects (e.g., Doctrine entities) may fail to serialize.
   • Fix: Use DTOs or manually serialize payloads:

     $job = new SendEmailJob();
     $job->setPayload(['user_id' => $user->id]); // Store only serializable data
     

4. Debugging

   • Enable Debug Mode:

     dv_evil_queue:
         debug: true
     

   • Log Levels: Configure monolog.logger.evil to debug for verbose logs.
   • Command-Line Tools:
     • List jobs: php bin/console evil-queue:list
     • Delete failed jobs: php bin/console evil-queue:delete-failed

5. Worker Stuck in "Busy" State

   • Cause: Long-running jobs or unhandled exceptions.
   • Fix: Implement JobInterface::run() with proper error handling and timeouts.

Tips

1. Environment-Specific Configs Override settings per environment (e.g., workers: 20 in production, 5 in staging).

2. Health Checks Add a route to check worker status:

   // src/Controller/QueueController.php
   public function healthCheck(JobDispatcherInterface $dispatcher): JsonResponse
   {
       return new JsonResponse([
           'workers' => $dispatcher->getWorkerCount(),
           'queue_size' => $dispatcher->getQueueSize(),
       ]);
   }
   

3. Testing

   • Mock the dispatcher in tests:

     $dispatcher = $this->createMock(JobDispatcherInterface::class);
     $dispatcher->expects($this->once())->method('dispatch');
     $this->controller->setDispatcher($dispatcher);
     

   • Use evil-queue:test-worker for isolated job testing.

4. Performance Tuning

   • Batch Processing: Process jobs in batches (e.g., 100 at a time) to reduce DB load.
   • Connection Pooling: Ensure doctrine.dbal.xmlrpc_connection is optimized for high concurrency.

5. Extending the Bundle

   • Custom Job Storage: Implement JobStorageInterface for alternative backends (e.g., Redis).
   • Hooks: Extend JobEvent classes to add custom events (e.g., JobPaused).

6. Security

   • Validate job payloads to prevent injection:

     public function run(): void
     {
         if (!is_numeric($this->payload['user_id'])) {
             throw new \RuntimeException('Invalid user ID');
         }
     }