Kafka Transport Laravel Package

Technical Evaluation

Architecture fit exoticcaengineering/kafka-transport remains a Laravel queue transport that maps the framework's queue abstraction (jobs, workers, retries, failed_jobs, Horizon-adjacent tooling) onto Kafka. But 0.2.5 is not a tuning release — it changes the delivery contract of the transport. "Commit on receive and leave the group before handling" moves the offset commit to before the job executes and removes the worker from the consumer group for the duration of processing. That is an at-most-once posture: once a message is fetched, its offset is durable, so a crash, OOM kill, deploy restart, or timeout mid-handling loses the job. It also breaks the implicit assumption Laravel's queue API is built on — that a job not acknowledged is retried. release(), $tries, backoff, and retry-until-failed all depend on the message becoming redeliverable; this release removes that lever. Second-order effect: the cooperative-rebalancing capability that made 0.2.2 credible now applies to a much shorter membership window (fetch, then leave), so most of that scale-out upside evaporates, and group membership churn plausibly increases. For a TPM this is a scope change disguised as a changelog line, and it needs a product-level sign-off on durability, not a library bump.

Integration feasibility Composer-installable, but the integration gate has moved from environment verification to a semantic proof. Three things must be demonstrated in a spike before any adoption decision:

1. Whether a job that throws still lands in failed_jobs and whether retry on a failed job can re-produce it, given the offset is already committed.
2. What "leave the group before handling" means granularly — per message, per batch/poll cycle, or per worker lifetime. Per-message leave/rejoin would be catastrophic for throughput and would trigger near-continuous rebalances; per-poll-cycle is tolerable but still means the group is never stably held during work.
3. Whether in-flight messages fetched but not yet committed are dropped on shutdown, and whether the package exposes any hook to drain before leaving. Note the version gap: we have 0.2.2 notes and 0.2.5 notes but not 0.2.3 or 0.2.4. A behavioral change of this magnitude shipped as a patch-level bump, with no migration notes, no documented rollback, and two intermediate releases we cannot inspect, is itself a finding.

Technical risk

• Data loss by design. At-most-once is a legitimate pattern, but it is the opposite of what most teams assume when they point Laravel's queue at Kafka. Silent job loss under load, deploys, or node eviction is the default outcome.
• Retry semantics inverted. $tries, backoff, release(), and failed-job retry all assume at-least-once. If the transport cannot redeliver, retries become no-ops or throw — an unknown that must be tested, not inferred.
• Ordering and partitioning. Leaving the group before handling means the worker no longer holds the partition it is processing. Per-partition ordering guarantees that teams may rely on (keyed jobs) can be broken by another consumer picking up the next offset while the first job is still running.
• Rebalance behavior is now untested territory. Constant leave/rejoin cycles from many replicas can produce rebalance churn, longer assignment latency, and group.initial.rebalance.delay.ms amplification — the exact failure class the 0.2.2 feature was meant to soothe.
• Still pre-1.0, single-org OSS, patch-level breaking change, incomplete public changelog. Supply-chain risk has gone up, not down.
• Documentation debt is now safety-relevant. A one-line note is insufficient for a durability change; there is no stated compatibility matrix, no config reference, and no stated default.

Key questions

1. Is commit-on-receive unconditional in 0.2.5, or behind a config flag? Exact key and default.
2. What is the failure path for a job that throws — is it recorded in failed_jobs, and can it be retried/re-produced? Does release() throw, no-op, or duplicate?
3. Granularity of "leave the group before handling": per message, per poll batch, or once? Any max.poll.interval.ms implications?
4. Are in-flight, uncommitted messages drained on SIGTERM, or dropped? Is there a graceful-shutdown contract for Kubernetes/Supervisor?
5. What changed in 0.2.3 and 0.2.4? Why is a delivery-semantics change a patch release, and is a revert to 0.2.4 supported?
6. Does partition ordering survive the leave-before-handle window, and is that documented?
7. Any guidance on idempotency, DLQ, or poison-message handling under at-most-once?

Integration Approach

Stack fit Laravel 9+/PHP 8.x with ext-rdkafka, containerized workers, Kafka 2.4+ (verify against actual constraints). 0.2.5 fits only workloads where losing an individual job is acceptable — metrics flushes, cache warmers, notification fan-out with its own retry, telemetry. It fits badly for financial postings, state transitions, email delivery with no upstream reconciliation, or anything the team currently trusts Laravel retries to make durable. The decision is per-queue, not per-application, and it should be documented as such.

Migration path

1. Do not roll 0.2.5 forward on the strength of the changelog. Hold production on the last known at-least-once version (0.2.4 pending verification) and pin exactly.
2. Build a throwaway spike: one topic, three replicas, a job that throws, a job that is SIGKILLed mid-run, a rolling restart. Record what is lost, what lands in failed_jobs, and what rebalances.
3. Classify existing queues by durability requirement; produce an explicit list of queues that may run at-most-once.
4. If adopted, adopt only for the classified-low-value queues, on separate topics and separate worker deployments from durable queues.
5. Canary one low-value queue through a full deploy cycle before widening.

Compatibility Treat 0.2.5 as an all-or-nothing cutover per consumer group, as before, but now for a different reason: mixing members means mixing commit-timing semantics within a group, which makes loss behavior non-deterministic and unobservable. Broker and librdkafka floors still need verification, and group.initial.rebalance.delay.ms plus session/heartbeat tuning need re-checking against the new membership churn profile — previous tuning is likely wrong.

Sequencing Verify 0.2.3/0.2.4 contents → confirm flag/default → durability spike with forced failures → stakeholder sign-off on at-most-once for named queues → queue classification → separate worker pools → canary on the lowest-value queue → monitor loss indicators for one full deploy cycle → expand or abandon. Rollback is not a redeploy: because committed offsets cannot be un-committed, "rollback" to 0.2.4 restores redelivery only for messages not yet received. Lost jobs are unrecoverable and must be reconciled from an upstream source of truth. That reconciliation path should exist before rollout, not after.

Operational Impact

Maintenance Code cost stays low; governance cost rises sharply. You now own a documented decision about which jobs may be lost, and that decision must be revisited whenever a new job class is added to an affected queue — an ongoing, easy-to-forget obligation. Version pinning is still mandatory, librdkafka revalidation continues, and the single-contributor concentration on delivery-path code is now a bigger liability than before, since a fork or vendor patch would touch commit semantics rather than configuration.

Support No vendor, no LTS. On-call must now handle a failure mode Laravel teams are not trained for: absence of a job. Existing Laravel reflexes — check failed_jobs, retry, replay the queue — will not work. The runbook needs a "missing job" procedure with an upstream reconciliation source, plus explicit escalation for rebalance churn under the new leave/rejoin pattern. The team should also maintain a pinned known-good version and a tested revert.

Scaling The 0.2.2 upside is largely neutralized. Cooperative rebalancing mainly helps when consumers hold partitions during long work; leaving the group before handling means the worker is a member only while fetching, so assignment churn happens more often and matters less per event. Throughput is now dominated by fetch/commit round-trips and group join cost rather than job duration, and long-running jobs no longer risk max.poll.interval.ms eviction — which is likely the motivation for the change, and a real benefit for slow jobs. Net: better for long, unreliable-duration jobs; worse for steady high-throughput queue workloads that relied on stable group membership.

Failure modes

• Worker crash, OOM, or SIGKILL mid-job → committed offset, job permanently lost, no failed_jobs entry by default. This is the dominant new risk.
• Throwing job → offset already committed; retry path either no-ops or, worse, silently duplicates while the original is dropped.
• Rolling deploy → in-flight jobs lost per pod unless drain-on-shutdown exists.
• High replica counts → leave/rejoin churn, repeated assignment latency, possible rebalance storms from aggressive session/heartbeat settings.
• Ordering breakage → a second consumer processes offset N+1 while N is still running, violating keyed ordering.
• Config drift → one environment at-most-once, another at-least-once, with identical-looking dashboards.
• Revert-after-loss → restoring 0.2.4 recovers nothing already committed; teams will discover this during an incident if it is not written down.

Ramp-up Higher than the 0.2.2 baseline. The Laravel wiring is still an afternoon, but the team must now internalize at-most-once reasoning: what "committed" means, why retries are a transport decision rather than a framework one, how to instrument for missing work (reconciliation counts, expected-vs-processed volume), and why partition ordering depends on holding assignment. Budget an enablement session, a written queue-classification doc, and a reconciliation runbook before any production use — and treat the missing 0.2.3/0.2.4 changelogs and absent migration guidance as documentation the team must backfill and maintain itself.