Queue Laravel Package

Technical Evaluation

Architecture Fit

• Use Case Alignment: The tarantool/queue package is designed for asynchronous task processing with distributed job queues, leveraging Tarantool (a high-performance in-memory database) as the backend. It fits well in architectures requiring:
  • High-throughput job processing (e.g., batch processing, event-driven workflows).
  • Low-latency task execution (Tarantool’s in-memory storage reduces I/O bottlenecks).
  • Scalable worker pools (distributed job consumption via multiple workers).
  • Job persistence with retry/failure handling (critical for reliability).
• Anti-Patterns:
  • Not ideal for simple cron jobs (overhead of Tarantool setup may not justify gains).
  • Tight coupling to Tarantool (if your stack avoids Tarantool, alternatives like Redis or database-backed queues may be simpler).
  • Stateful workers (if jobs require complex state management, consider frameworks like Laravel Queues with Redis).

Integration Feasibility

• Laravel Compatibility:
  • The package is PHP-native and can integrate with Laravel via:
    • Custom queue driver (extend Laravel’s queue system to use Tarantool).
    • Direct API usage (bypass Laravel queues entirely for high-performance needs).
  • Dependencies:
    • Requires Tarantool client library (tarantool/tarantool-php) and a running Tarantool instance.
    • No native Laravel service provider or queue worker integration (manual setup required).
• Data Model:
  • Jobs are stored as key-value pairs in Tarantool, with metadata for retries, delays, and failures.
  • Serialization: Uses PHP’s serialize() by default (may need customization for complex payloads).

Technical Risk

Key Questions

1. Why Tarantool?
   • Does the team have experience with Tarantool, or is this a greenfield decision?
   • How does its performance compare to Redis/DB queues for your workload?
2. Laravel Integration Depth
   • Will you replace Laravel’s queue system entirely, or just add Tarantool as a driver?
   • How will job payloads (e.g., Laravel notifications, commands) serialize/deserialize?
3. Operational Trade-offs
   • What’s the backup/recovery strategy for Tarantool-stored jobs?
   • How will monitoring (e.g., job stuck/failure rates) be implemented?
4. Alternatives
   • Have Redis-based solutions (e.g., predis, laravel-queue-redis) been ruled out?
   • Would a database-backed queue (e.g., laravel-queue-database) suffice?

Integration Approach

Stack Fit

• Best For:
  • High-performance PHP apps needing sub-millisecond job latency.
  • Microservices with distributed workers (Tarantool’s replication supports multi-node setups).
  • Event sourcing/CQRS where job ordering and persistence are critical.
• Stack Constraints:
  • Not suitable for serverless (Tarantool requires persistent connections).
  • Avoid if: Your stack is Redis-heavy or uses managed queue services (e.g., AWS SQS).

Migration Path

1. Proof of Concept (PoC)
   • Set up a single-node Tarantool instance and test basic job enqueue/dequeue.
   • Compare latency/throughput vs. current queue (e.g., Redis).
2. Laravel Integration Layer
   • Option A: Extend Laravel’s queue system by creating a TarantoolQueue driver.
     • Implement Illuminate\Contracts\Queue\Queue interface.
     • Use tarantool/tarantool-php for direct calls.
   • Option B: Bypass Laravel queues entirely and use the package directly in workers.
3. Data Migration
   • If migrating from another queue (e.g., Redis), write a script to export jobs and reimport into Tarantool.
   • Example schema for jobs:

     -- Tarantool space for jobs
     box.schema.space.create('jobs', {if_not_exists = true})
     box.schema.space:format({
       {name = 'id', type = 'unsigned'},
       {name = 'payload', type = 'string'},
       {name = 'attempts', type = 'unsigned'},
       {name = 'reserved_at', type = 'unsigned'},
       {name = 'available_at', type = 'unsigned'},
     })
     

Compatibility

• Laravel Versions: Tested with Laravel 10+ (PHP 8.1+). May need adjustments for older versions.
• Tarantool Compatibility: Requires Tarantool 2.10+ (check for Lua/PHP API stability).
• Worker Patterns:
  • Supports multiple workers (each connects to Tarantool).
  • No built-in supervisor (use Laravel’s queue:work or a custom process manager).

Sequencing

1. Phase 1: Core Integration
   • Implement Tarantool queue driver for Laravel.
   • Test job lifecycle (enqueue, process, retry, fail).
2. Phase 2: Scaling
   • Set up Tarantool replication for HA.
   • Optimize worker connection pooling.
3. Phase 3: Observability
   • Add job metrics (e.g., Prometheus exporter for Tarantool).
   • Implement DLQ for failed jobs.
4. Phase 4: Rollout
   • Canary deploy: Route non-critical jobs to Tarantool first.
   • Monitor for connection leaks or performance regressions.

Operational Impact

Maintenance

• Tarantool Management:
  • Requires Lua scripting for schema changes or custom logic.
  • No ORM: Direct key-value operations mean manual index management.
• Laravel Integration:
  • Custom queue driver may need updates if Laravel’s queue contract changes.
  • Dependency bloat: Adding tarantool/tarantool-php increases deployment size.
• Upgrade Path:
  • Tarantool upgrades may break PHP client compatibility.
  • Laravel version upgrades could require driver adjustments.

Support

• Community/Ecosystem:
  • Limited Laravel-specific support (package is PHP/Tarantool-focused).
  • Tarantool community is smaller than Redis; troubleshooting may require deeper expertise.
• Debugging:
  • Job failures: Logs must include Tarantool connection errors and Lua script failures.
  • Worker health: Monitor for stuck connections or memory leaks in PHP workers.
• Vendor Lock-in:
  • Custom Tarantool schemas or Lua logic may be hard to migrate away from.

Scaling

• Horizontal Scaling:
  • Workers: Scale by adding more PHP workers (stateless).
  • Tarantool: Scale by adding replicas (read scaling) or shards (write scaling).
• Performance Bottlenecks:
  • Network latency: Tarantool clients must be co-located with the database for low latency.
  • Connection overhead: Each worker needs a persistent Tarantool connection.
• Throughput:
  • Benchmark: Test with 10K+ jobs/sec to validate Tarantool’s limits.
  • Alternatives: If Tarantool hits limits, consider Tarantool + Redis hybrid (e.g., use Redis for metadata).

Failure Modes