Queue Laravel Package

Technical Evaluation

Assessment refreshed for tarantool/queue v0.11.0 (prior baseline: v0.10.0).

Architecture Fit

• Use Case Alignment: The tarantool/queue package remains an asynchronous task processing library backed by Tarantool (in-memory, persistent, replicated). It still fits architectures requiring:
  • High-throughput job processing (batch, event-driven workflows).
  • Low-latency task execution (in-memory storage avoids I/O bottlenecks).
  • Scalable worker pools (distributed consumption).
  • Job persistence with retry/failure handling.
• New in v0.11.0 — architecture-relevant capabilities:
  • Typed queue APIs formalize payload contracts (DTO-style producers/consumers). This moves the package closer to "framework-grade" ergonomics and reduces the bespoke glue we previously flagged as the main integration tax.
  • Dynamically named subqueues with subqueuettl make tenant-, priority-, or time-bucketed queues a first-class primitive with automatic expiry — previously this required hand-written Lua schema work and our own cleanup jobs.
  • Bounded queues with Queue::put() returning null when full introduces a native backpressure signal, which the package did not previously expose.
• Anti-Patterns (unchanged):
  • Still not ideal for simple cron jobs (Tarantool operational overhead).
  • Still tightly coupled to Tarantool as a backend.
  • Still not a drop-in Laravel queue driver; no service provider, no queue:work integration.
• Anti-Patterns (new):
  • Unbounded subqueue cardinality: dynamic names make it trivially easy to create a subqueue per tenant/user/request-id. This is now a design decision we must govern, not an implementation detail.
  • Ignoring put()'s nullable return converts a deliberate backpressure mechanism into silent job loss.

Integration Feasibility

• Laravel Compatibility:
  • Integration paths are unchanged: custom queue driver (implement Illuminate\Contracts\Queue\Queue) or direct API usage bypassing Laravel queues.
  • Version floor moved: v0.11.0 requires PHP 8.2+ (was 8.1+). Laravel 10 supports PHP 8.1, so a Laravel 10 estate on 8.1 is pinned to v0.10.x until the runtime is upgraded. Laravel 11+ (PHP 8.2 minimum) is unaffected.
• Dependencies:
  • Requires tarantool/client ^0.11 (supersedes the previously referenced tarantool/tarantool-php client). Client and queue library must now be upgraded in lockstep — this is a new co-dependency constraint.
  • Still requires a running Tarantool instance; no native Laravel service provider.
• Data Model:
  • Jobs remain key-value entries in a Tarantool space with retry/delay/failure metadata.
  • Serialization: typed APIs reduce reliance on ad-hoc serialize() handling for complex payloads; the escape hatch for arbitrary payloads still exists and must be governed.
• Backpressure contract:
  • put() returning null means "not enqueued." Any driver wrapper must map this to Laravel's push() false so callers can branch, and any caller that treats a job handle as guaranteed will break at runtime rather than at compile time.

Technical Risk

Key Questions

1. Runtime readiness — Is the fleet on PHP 8.2+? If not, which services stay pinned to v0.10.x, and what is the upgrade date?
2. Overflow policy — When a bounded queue returns null, do we fail the request, spill to a fallback, or retry? Who is alerted, and how do we detect dropped jobs?
3. Subqueue governance — What is the naming scheme, expected cardinality ceiling, and retention/TTL per subqueue class? Does TTL expiry risk discarding pending work?
4. Tarantool server support — Which server versions does v0.11.0 actually test against, given the tarantoolctl removal in CI? Is Tarantool 3.x supported?
5. Typed API adoption — Do we adopt typed payloads for new job types only, or retrofit existing payloads? What is the migration cost for in-flight jobs serialized the old way?
6. Why Tarantool? — Unchanged: does the throughput/latency win over Redis or a database queue still justify the operational surface, now that Redis-backed backpressure and sub-queue tooling are mainstream?

Integration Approach

Stack Fit

• Best For (unchanged): high-performance PHP apps needing sub-millisecond job latency; microservices with distributed workers; event sourcing/CQRS where ordering and persistence matter.
• Newly best for: multi-tenant and priority-partitioned workloads, where per-tenant or per-priority subqueues with TTL replace hand-rolled partitioning and cleanup jobs.
• Stack Constraints:
  • PHP 8.2+ is now a hard prerequisite (v0.11.0).
  • Still unsuitable for serverless (persistent connections required).
  • Still a poor fit if the stack is Redis-heavy or uses managed queues (SQS).

Migration Path

1. Prerequisite gate — Confirm PHP 8.2+ on target hosts and Laravel major compatibility; document the v0.10.x pin and exit date for any 8.1 hosts.
2. Proof of Concept (PoC) — Single-node Tarantool; benchmark latency/throughput vs. current queue. Extend the PoC to cover v0.11.0 specifics: bounded-queue full behavior, put() null handling, and subqueue TTL expiry with pending jobs.
3. Laravel Integration Layer
   • Option A: TarantoolQueue driver implementing Illuminate\Contracts\Queue\Queue; map put() === null → push() === false.
   • Option B: Direct package usage in workers, bypassing Laravel queues.
4. Payload strategy — Adopt typed APIs for new job types; keep a compatibility path for legacy serialized payloads already in the queue at cutover.
5. Data Migration — Drain or export jobs from the existing queue; reimport into Tarantool. Existing space schema remains valid as a starting point.

Compatibility

• PHP: 8.2+ required (v0.11.0). v0.10.x remains the option for PHP 8.1.
• Laravel: Practically Laravel 10 on PHP 8.2+, or Laravel 11+. Re-validate older majors before assuming support.
• Tarantool client: tarantool/client ^0.11, upgraded in lockstep with the queue library.
• Tarantool server: Prior assessment assumed 2.10+; the CI move away from tarantoolctl suggests newer tooling. Treat the tested server matrix as unverified until confirmed.
• Worker Patterns: Multiple workers supported; no built-in supervisor — continue using queue:work or a custom process manager.

Sequencing

1. Phase 0: Runtime + dependency alignment — PHP 8.2+, tarantool/client ^0.11, lockfile pins, integration test on enqueue/dequeue.
2. Phase 1: Core integration — Tarantool queue driver with nullable-put() mapping; test lifecycle (enqueue, process, retry, fail, full-queue rejection).
3. Phase 2: Subqueue design — Approve naming convention, cardinality budget, and TTL values; roll out one subqueue class (e.g., per-tenant) before generalizing.
4. Phase 3: Scaling — Replication/HA, connection pooling, worker concurrency tuning.
5. Phase 4: Observability — Metrics for throughput, null-return rate, subqueue count, TTL expiries, DLQ depth.
6. Phase 5: Rollout — Canary non-critical jobs; watch for connection leaks, dropped-job alarms, and version skew across workers.

Operational Impact

Maintenance

• Upgrade cadence is the dominant cost. v0.11.0 bundled a runtime floor bump, a client major-minor bump, a behavioral return-type change, and new APIs into one release. Budget a planned integration cycle per minor version; avoid unmanaged composer update.
• PHP 8.2 upgrade work is now on the critical path and should be treated as a separate, schedulable workstream.
• Reduced manual toil: subqueuettl removes the custom cleanup cron we previously would have written; typed APIs reduce bespoke serialization glue.
• New toil: subqueue naming/cardinality governance, TTL value tuning, and overflow-policy maintenance.
• Tarantool management unchanged: Lua scripting for schema/custom logic, no ORM, manual index management.
• Upgrade path: client/server version drift can break the PHP client; Laravel queue-contract changes can break the driver.

Support

• Community: Still limited Laravel-specific support; Tarantool community smaller than Redis. v0.11.0's new contributors touched CI only, so bus factor on the library logic remains concentrated — a standing continuity risk.
• Debugging:
  • Job failures require correlating Tarantool connection errors, Lua failures, and PHP worker logs.
  • New: "job never ran" is now a plausible symptom of either TTL expiry or a null-return drop — both look like silence, not errors. Instrument both explicitly.
• Vendor lock-in: Custom schemas, subqueue naming schemes, and Lua logic remain hard to migrate away from.

Scaling

• Workers: Horizontal by adding stateless PHP workers (each holding a persistent Tarantool connection).
• Subqueues as a scaling lever: per-tenant/per-priority isolation is now cheap to implement, and TTL prevents orphaned partitions from growing forever.
• New scaling limits to watch:
  • Subqueue cardinality — space/index growth and per-queue stats overhead.
  • Bounded-queue sizing — too small causes legitimate backpressure under burst; too large delays the signal until the incident is already severe.
  • Connection count — scales linearly with workers; pool or cap before adding nodes.
• Throughput: re-benchmark at 10K+ jobs/sec; typed APIs may shift serialization cost profiles slightly.

Failure Modes