Kafka Transport Laravel Package

Getting Started

Release 0.2.5 changes delivery semantics, not just internals: the consumer now commits the offset on receive and leaves the consumer group before the job is handled. Read that sentence twice before you write consumer code — Kafka will no longer redeliver a message because your job threw an exception. Everything below flows from that.

1. Install and check the extension. The package wraps a native Kafka client, so confirm ext-rdkafka (or your configured client binding) is present in both your app and your worker/container images. composer require exoticcaengineering/kafka-transport, then php artisan vendor:publish --tag=kafka-transport-config.
2. Look first at the published config, not the README. You want broker/bootstrap.servers, consumer group.id, topic mapping, the partition assignment strategy (0.2.2's cooperative rebalancing — grep for assignment / rebalance / cooperative), and any new commit/leave toggles introduced in 0.2.5. Confirm the exact key names and accepted values against the version actually installed in composer.lock.
3. Wire a queue connection in config/queue.php pointing at the package's connector, with topic and consumer group per connection. One connection per consumer-group/topic pairing.
4. First use case: dispatch a job onto that connection (MyJob::dispatch($payload)->onConnection('kafka')), run the consumer command in a separate process, confirm the job executes. Then do the check that actually matters in 0.2.5: kill -9 the worker while the job is running and verify the message is not redelivered. That single experiment tells you whether your failure handling is ready.

Do all of this against a single-broker local cluster with one consumer before staging. Rebalance behaviour is invisible at that scale, and it's still what the 0.2.2 → 0.2.5 line changed most.

Implementation Patterns

Producer side — unchanged. Treat it as a queue, not an event bus. Dispatching jobs through the queue connection inherits Laravel's serialization, retries, and failed_jobs handling. Reserve raw producer usage for publishing integration events you don't own consuming.

Consumer side — Kafka is now a transport, not a retry mechanism. Because the offset is committed before handling, redelivery on failure is gone. Your retry story moves entirely into Laravel: job $tries/backoff, failed_jobs, queue:retry, and alerting on failed-job volume. If you previously relied on "the message comes back if the job fails," that behaviour is now different — build the compensating path explicitly, and consider a dead-letter table or topic written from failed().

Concurrency is decoupled from partition count — deliberately. Workers join, take an assignment, commit, leave, then process. That's the point of the release: a long-running job no longer sits inside the group holding a partition, so it can't be evicted by max.poll.interval.ms or block a rebalance. You can now run more workers than partitions. Size workers by job throughput and downstream capacity, not by num.partitions. But see the churn gotcha below before you autoscale wildly.

Ordering is no longer guaranteed across a partition. Two messages from the same partition can be handled by different workers concurrently. If any of your topics rely on per-key ordering, this release breaks that assumption. Either route ordered work through a single-worker group, add a serialization key with your own locking, or keep ordered topics on a pre-0.2.5 version. Decide per topic, not globally.

Revoke handlers get simpler — keep them partition-scoped anyway. Cooperative assignment from 0.2.2 still applies, and revocations can arrive as a delta, sometimes twice. Any onPartitionsRevoked-style hook must operate strictly on the partitions passed in. The good news: because handling happens outside the group, your in-flight work is no longer at the mercy of a rebalance, so these hooks shrink to offset bookkeeping.

Roll out as a group replacement, not a rolling deploy. Old workers (commit-after-handle) and new workers (commit-on-receive) in the same group produce mixed delivery semantics and confusing offset movement. Either scale the group to zero, deploy, and scale back, or stand up a new consumer group name, let it drain, and retire the old one.

// config/kafka-transport.php (verify exact keys against your install)
'consumers' => [
    'default' => [
        'group_id' => env('KAFKA_GROUP_ID'),
        'topics'   => ['orders'],
        'assignment_strategy' => env('KAFKA_ASSIGNOR', 'cooperative-sticky'),
    ],
],

Testing. Queue::fake() / Bus::fake() remains your default for producer assertions. Add to your opt-in integration suite: commit-on-receive behaviour (kill mid-job, assert no redelivery), the failed() path firing, and topic/group naming. Those are exactly the behaviours fakes cannot validate and exactly what 0.2.5 changed.

Gotchas and Tips

• This is effectively at-most-once from Kafka's perspective. Commit happens before handling; a crash between commit and completion loses the message. If your workloads can't tolerate loss, this release is a behaviour change you must design around, not a drop-in upgrade.
• Kafka lag monitoring goes blind. Consumer group lag will hover near zero because offsets advance on receive. Backlog now lives in your application, not in the broker. Add queue-depth/failed-job metrics, or you will discover a backlog only when the downstream complains.
• Consumer-group churn goes up sharply. Every message means a join, assignment, commit, and leave. High-volume topics can generate significant group-coordinator traffic and rebalance noise. Watch broker logs and coordinator CPU; tune group.initial.rebalance.delay.ms and consider batching or longer-lived consumption patterns if churn becomes the bottleneck.
• Upgrade is not rolling-safe. Mixed commit semantics inside one group produce offset weirdness and silent loss. Blue/green the group, or stop the world.
• Idempotency is still mandatory, just for a different reason: crash-before-commit still redelivers, and the commit/handle boundary is not atomic with your side effects. Make consumers idempotent by message key and log the partition/offset you committed at receive time — that log is now your only reliable link between a Kafka message and a handled job.
• Long-running jobs are finally safe, but graceful shutdown still matters. Jobs are no longer bounded by max.poll.interval.ms. You still need php artisan queue:restart on deploy and a shutdown window so in-flight work finishes rather than being killed mid-handle (which, post-0.2.5, means lost work).
• Client version floor. Cooperative-sticky needs a reasonably modern librdkafka in the underlying client. If cooperative-sticky is silently ignored or the group falls back, check the client version in the worker image before suspecting the package.
• Memory in long-running workers. Still relevant for the consumer process even though it detaches from the group between messages — watch accumulating Eloquent models and container-bound references, keep the consumer connection lean, and avoid resolving heavy singletons per message without releasing them.
• Debugging path: broker-side kafka-consumer-groups --describe --group <id> for assignment and (now misleading) lag; worker logs for the negotiated assignor, commit-on-receive offset, and revoke callbacks; failed_jobs plus your own metrics for the real backlog; and the published config to confirm which strategy shipped.
• Extension points: the published config and the connector registration in config/queue.php are the intended seams. Prefer subclassing the connector/consumer command over editing vendor code, so the next patch — which may change delivery semantics again — is a clean composer update. Pin the version and read the diff before bumping.