Laravel Sqs Fifo Queue Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require shiftonelabs/laravel-sqs-fifo-queue
   

   • No manual service provider registration required in Laravel 9+ (auto-discovery).

2. Configure SQS FIFO Queue: Add the driver to your .env:

   QUEUE_CONNECTION=fifo_sqs
   

   Configure SQS credentials and FIFO queue URL in config/queue.php:

   'connections' => [
       'fifo_sqs' => [
           'driver' => 'fifo_sqs',
           'key' => env('AWS_ACCESS_KEY_ID'),
           'secret' => env('AWS_SECRET_ACCESS_KEY'),
           'prefix' => env('SQS_QUEUE_PREFIX', 'https://sqs.region.amazonaws.com/'),
           'queue' => env('SQS_FIFO_QUEUE_URL', 'your-fifo-queue-url.fifo'),
           'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
           'options' => [
               'FifoQueue' => true,
               'MessageGroupId' => env('SQS_MESSAGE_GROUP_ID', 'default-group'),
           ],
       ],
   ],
   

3. First Use Case: Dispatch a job with a FIFO queue:

   use App\Jobs\ProcessOrder;
   
   ProcessOrder::dispatch($orderId)->onConnection('fifo_sqs');
   

Implementation Patterns

Core Workflows

1. Job Dispatching with Message Grouping:

   • Use MessageGroupId to control ordering and deduplication:

     ProcessOrder::dispatch($orderId)
         ->onConnection('fifo_sqs')
         ->withGroupId('order-group-' . $orderId); // Custom group ID
     

2. Handling FIFO-Specific Features:

   • Deduplication: Set DeduplicationId in job payload or via withDeduplicationId():

     ProcessOrder::dispatch($orderId)->withDeduplicationId($orderId);
     

   • Visibility Timeout: Override in config/queue.php:

     'fifo_sqs' => [
         'options' => [
             'VisibilityTimeout' => 300, // 5 minutes
         ],
     ],
     

3. Batch Processing:

   • Leverage FIFO’s SendMessageBatch for bulk operations:

     $batch = new \Aws\Sqs\Message\SendMessageBatch();
     $batch->setMessages([
         new \Aws\Sqs\Message\SendMessage([
             'Id' => '1',
             'MessageBody' => json_encode(['job' => 'ProcessOrder', 'data' => $orderId]),
             'MessageGroupId' => 'batch-group',
             'MessageDeduplicationId' => $orderId,
         ]),
     ]);
     $sqs->sendMessageBatch($batch);
     

4. Queue Listeners:

   • Process jobs in order using Laravel’s queue:listen:

     php artisan queue:listen fifo_sqs --sleep=3 --tries=3
     

   • For Lumen, use php artisan queue:listen fifo_sqs.

Integration Tips

• AWS SDK Compatibility: Ensure your aws/aws-sdk-php version is compatible (tested with ^3.0).
• Environment Awareness: Use env() for dynamic queue URLs/regions in config/queue.php.
• Monitoring: Integrate with AWS CloudWatch for FIFO queue metrics (e.g., ApproximateNumberOfMessagesVisible).

Gotchas and Tips

Pitfalls

1. Message Group ID Collisions:

   • FIFO queues require unique MessageGroupId per logical sequence. Reusing IDs may cause out-of-order processing.
   • Fix: Use job-specific prefixes (e.g., user-{id}-group).

2. Deduplication Scope:

   • MessageDeduplicationId must be unique within the group ID for 5 minutes. Reusing IDs prematurely will silently drop messages.
   • Fix: Use UUIDs or timestamps for deduplication IDs.

3. Batch Limitations:

   • SQS FIFO batches have a 10-message limit and 25KB payload limit per message.
   • Fix: Split large batches or use standard SQS for non-critical bulk jobs.

4. Visibility Timeout:

   • Long-running jobs may exceed the default VisibilityTimeout (30 seconds), causing premature retries.
   • Fix: Increase timeout in config or use release() to extend visibility manually.

5. Laravel Job Serialization:

   • FIFO queues may fail if job payloads exceed SQS’s 256KB limit (including metadata).
   • Fix: Offload large data to S3 or use database queue for heavy payloads.

Debugging

• Check Queue URL: Verify SQS_FIFO_QUEUE_URL in .env matches your AWS console URL (must end with .fifo).
• AWS Credentials: Test SQS access with:

  aws sqs get-queue-url --queue-name your-queue.fifo --region us-east-1
  

• Logs: Enable AWS SDK debugging:

  'options' => [
      'debug' => true,
      'logger' => new \Monolog\Logger('sqs'),
  ],
  

Extension Points

1. Custom Job Payloads: Override toSqsArray() in your job to include FIFO-specific metadata:

   public function toSqsArray()
   {
       return array_merge(parent::toSqsArray(), [
           'MessageGroupId' => 'custom-group',
           'MessageDeduplicationId' => $this->deduplicationId,
       ]);
   }
   

2. Middleware for FIFO: Add queue-specific middleware in App\Providers\QueueServiceProvider:

   public function boot()
   {
       Queue::before(function (JobProcessing $event) {
           if ($event->connectionName === 'fifo_sqs') {
               // Add FIFO-specific logic (e.g., group ID validation)
           }
       });
   }
   

3. Fallback to Standard SQS: Dynamically switch drivers based on conditions:

   $queueConnection = config('app.env') === 'production' ? 'fifo_sqs' : 'sync';
   ProcessOrder::dispatch()->onConnection($queueConnection);
   

4. Dead-Letter Queues (DLQ): Configure DLQ in config/queue.php:

   'fifo_sqs' => [
       'options' => [
           'QueueOwnerAWSAccountId' => env('AWS_ACCOUNT_ID'),
           'RedrivePolicy' => json_encode([
               'deadLetterTargetArn' => env('SQS_DLQ_ARN'),
               'maxReceiveCount' => 3,
           ]),
       ],
   ],