Snsqs Laravel Package

Getting Started

Minimal Setup

1. Install the package:

   composer require enqueue/snsqs
   

2. Configure AWS credentials (via ~/.aws/credentials, environment variables, or Laravel's .env):

   AWS_ACCESS_KEY_ID=your_key
   AWS_SECRET_ACCESS_KEY=your_secret
   AWS_REGION=us-east-1
   

3. Set up a queue connection in Laravel (in config/queue.php):

   'connections' => [
       'snsqs' => [
           'driver' => 'snsqs',
           'queue' => env('SQS_QUEUE_URL', 'https://sqs.us-east-1.amazonaws.com/1234567890/my-queue'),
           'region' => env('AWS_REGION', 'us-east-1'),
           'options' => [
               'visibilityTimeout' => 60, // seconds
           ],
       ],
   ],
   

4. Dispatch a job:

   use Illuminate\Support\Facades\Queue;
   
   Queue::connection('snsqs')->push(new MyJob);
   

5. Consume messages (via Laravel's queue worker):

   php artisan queue:work snsqs --queue=my-queue
   

First Use Case

Use snsqs for decoupling long-running tasks (e.g., image processing, notifications) or scaling background jobs in a serverless environment. Example:

// Dispatch a delayed job
Queue::connection('snsqs')->later(now()->addMinutes(5), new SendEmailJob($user));

Implementation Patterns

Workflows

1. Job Dispatching

   • Use Queue::connection('snsqs')->push() for synchronous-like behavior.
   • Leverage later() for delayed execution (SQS supports up to 15-minute delays).
   • Batch processing: Dispatch multiple jobs in a loop (SQS handles concurrency).

2. Message Consumption

   • Run Laravel's queue worker with the snsqs connection:

     php artisan queue:work snsqs --queue=high-priority
     

   • Scaling: Deploy multiple workers across EC2 instances or Lambda functions (SQS distributes messages evenly).

3. Error Handling

   • Failed jobs are retried automatically (configurable in config/queue.php under max_attempts).
   • Use Queue::failed() to inspect failed jobs and reprocess manually:

     $failed = Queue::connection('snsqs')->failed();
     foreach ($failed as $job) {
         $job->release(); // Requeue
     }
     

4. Integration with AWS SNS

   • Publish messages to SNS topics and subscribe SQS queues to them for fan-out architectures:

     use Aws\Sns\SnsClient;
     
     $sns = new SnsClient([...]);
     $sns->publish([
         'TopicArn' => 'arn:aws:sns:us-east-1:1234567890:my-topic',
         'Message' => json_encode(['job' => 'SendEmailJob']),
     ]);
     

Laravel-Specific Tips

• Queue Middleware: Apply middleware to jobs (e.g., logging, rate limiting) via HandleJobsMiddleware in app/Http/Kernel.php.
• Queue Events: Listen to job.processing, job.failed, etc., for observability:

  event(new JobProcessing($job));
  

• Dynamic Queues: Use Queue::connection('snsqs')->on('dynamic-queue-'.$userId)->push($job).

Gotchas and Tips

Pitfalls

1. Visibility Timeout

   • If a worker crashes before completing a job, the message re-appears in the queue after the visibilityTimeout (default: 30s).
   • Fix: Increase visibilityTimeout in the connection config for long-running jobs (max: 12 hours).

2. Message Size Limits

   • SQS enforces a 256KB message size limit. Serialize payloads efficiently (e.g., JSON) or use S3 for large data.
   • Workaround: Store large data in a database and pass only IDs in the queue message.

3. Region Mismatch

   • Ensure the SQS queue URL and AWS region in the config match. Example:

     'queue' => 'https://sqs.us-west-2.amazonaws.com/1234567890/my-queue', // Must match region
     'region' => 'us-west-2',
     

4. IAM Permissions

   • The AWS IAM user/role must have:
     • sqs:SendMessage, sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes.
     • For SNS integration: sns:Publish.
   • Debugging: Check CloudTrail logs if permissions fail.

5. Dead-Letter Queues (DLQ)

   • Configure a DLQ in the SQS queue settings to capture unprocessable messages.
   • Laravel Setup: Add redrivePolicy to the queue URL:

     'queue' => 'https://sqs.us-east-1.amazonaws.com/1234567890/my-queue?AttributeName=RedrivePolicy&AttributeValue=...',
     

Debugging

• Check Queue Metrics: Use AWS CloudWatch to monitor ApproximateNumberOfMessagesVisible and NumberOfMessagesSent.
• Log Raw Messages: Temporarily log the raw payload in the job's handle() method:

  \Log::debug('SQS Message:', ['body' => $this->data]);
  

• Test Locally: Use LocalStack to mock SQS/SNS:

  docker run -it -p 4566:4566 localstack/localstack
  

Extension Points

1. Custom Serialization

   • Override the default JSON serialization by binding a custom QueueInterop\Transport\Serialization\SerializerInterface:

     $container->bind(SerializerInterface::class, function () {
         return new CustomSerializer();
     });
     

2. Message Attributes

   • Add metadata to messages using MessageAttributes:

     Queue::connection('snsqs')->push(new MyJob, 'default', [
         'Priority' => 10,
     ]);
     

3. Batch Consumption

   • Fetch multiple messages at once (up to 10) for efficiency:

     $context = new Context();
     $messages = $transport->receive($context, 5); // Receive 5 messages
     

4. Custom Context

   • Extend the QueueInterop\Transport\Context class to add request-specific data (e.g., correlation IDs):

     $context = new class extends Context {
         public $correlationId = '123';
     };