Kafka Bundle Laravel Package

Getting Started

Before writing any code, check the repo itself — this is a Symfony bundle, not a Laravel package. There is no extra.laravel.providers, no facade, no auto-discovery. Day one for a Laravel dev is a go/no-go decision:

• Symfony shop → composer require mgid/kafka-bundle, register in config/bundles.php, configure config/packages/mgid_kafka.yaml.
• Laravel shop → you have three realistic paths:
  1. Use mateusjunges/laravel-kafka or enqueue/rdkafka instead (native Laravel integration, actively maintained).
  2. Keep the bundle for a Symfony service you already run, and talk to it over HTTP/queue.
  3. Pull in only its non-bundle classes and wire them by hand in a ServiceProvider. This is the highest-effort, lowest-payoff option given the package's 0 stars / 0 dependents.

Prerequisites regardless of framework: ext-rdkafka (php-rdkafka) plus librdkafka installed in the image. Pin both — librdkafka version drift is a classic silent-breakage source.

# Debian/Ubuntu
RUN apt-get install -y librdkafka-dev && pecl install rdkafka && docker-php-ext-enable rdkafka

First use case worth wiring: publish one domain event (e.g. order.created) from an existing service, then consume it in a worker and log it. Don't start with a full event bus.

Minimal config shape (verify key names against the repo's DependencyInjection/Configuration.php — this is where you should look first, it's the real documentation):

mgid_kafka:
  brokers: "%env(KAFKA_BROKERS)%"   # "kafka-1:9092,kafka-2:9092"
  producers:
    default: { topic: orders }
  consumers:
    orders: { group: order-service }

Implementation Patterns

Producer as a thin port. Don't inject the bundle's producer deep into domain code. Wrap it:

interface EventPublisher { public function publish(string $topic, array $payload): void; }

final class KafkaEventPublisher implements EventPublisher
{
    public function __construct(private $producer) {}

    public function publish(string $topic, array $payload): void
    {
        $this->producer->send($topic, json_encode($payload + [
            'trace_id' => app(TraceId::class)->get(),
            'version'  => 1,
            'occurred_at' => now()->toIso8601String(),
        ]));
        $this->producer->flush(); // see gotchas
    }
}

Binding the bundle's producer service in Laravel means a manual $this->app->bind(...) in a provider, constructing the producer from RdKafka\Producer yourself. At that point you're mostly writing your own adapter anyway.

Envelope + versioning from day one. Kafka topics are immutable schemas in practice. Always JSON (or Avro/Protobuf) with version, trace_id, occurred_at, and type. Consumers switch on type, ignore unknown versions.

Outbox pattern for dual writes. The single biggest correctness win:

1. Write your row and an outbox row in one DB transaction.
2. A relay process (queue job or cron) polls the outbox and publishes.
3. Delete/mark-sent on success.

This decouples DB availability from broker availability and gives you replay.

Consumer as a dedicated worker. The bundle's consumer services are long-running poll loops; run them under Supervisor/systemd, not artisan queue:work. Typical shape:

while (true) {
    $message = $consumer->consume(1000); // yields or returns null on timeout
    if (!$message) { continue; }
    try {
        $this->handle(json_decode($message->payload, true));
        $consumer->commit($message);
    } catch (RetryableException $e) {
        $this->retryLater($message);
    }
}

Prefer manual commits over enable.auto.commit — you want at-least-once, and your handler must be idempotent (dedupe on event_id).

Wiring into a Laravel queue. A custom queue connector that maps dispatch() → publish() lets you keep the familiar ShouldQueue ergonomics while the actual transport is Kafka. Good bridge; the connector is ~80 lines.

Testing. Bind a FakeEventPublisher in tests that asserts on published payloads. Add one integration test with testcontainers/docker-compose for the real producer against a broker.

Gotchas and Tips

The Symfony/Laravel impedance mismatch is the headline gotcha. Symfony DI tags, compiler passes, and config/packages/*.yaml don't exist in Laravel. Expect to hand-wire services and re-implement any tagging logic yourself. This friction is the main reason not to adopt it in a Laravel app.

Maintenance risk is real. 0 stars, 0 dependents, score 0.16. Pin a specific commit, keep a fork, and read the source before adopting. Treat the bundle as a reference implementation of php-rdkafka, not as a safety net.

flush() semantics. flush() blocks. In a request path it can add latency spikes. Consider a small buffered publisher + flush($timeoutMs) at the end of the request, and always check the return code for QUEUE_FULL.

Delivery reports and errors are easy to skip. send() is async; without a delivery callback, produce errors vanish. Wire the callback and log topic/partition/offset/error.

Consumer group rebalancing. On partition reassignment your handler may see duplicate messages. Idempotency is not optional. Also: a slow handler blocks the whole poll loop and triggers max.poll.interval.ms eviction — keep handlers fast, push heavy work to a queue.

Long-running PHP pitfalls. Memory grows; gc_collect_cycles() periodically, restart workers on deploy. Handle SIGTERM to finish the current message and commit before exiting (Supervisor stopsignal=TERM, stopwaitsecs > your handler budget).

Serialization quirks. No schema registry integration — you own Avro/Protobuf wiring. JSON is simplest and usually right for an internal event bus.

Config passthrough. SASL/SSL options are librdkafka options; the bundle typically passes them verbatim. Check exact YAML key names against Configuration.php, and remember env var interpolation happens at container compile time in Symfony — %env()% vs %env(bool:...)% matters.

Debugging toolkit. Set librdkafka debug config (broker,protocol or all) temporarily; use kcat -b host -t topic -C -o beginning and kafka-consumer-groups.sh --describe to inspect lag and offsets. Most "messages aren't arriving" issues are ACL/topic-existence errors visible only in the delivery callback.

Extension points worth knowing. Symfony's tagged services / compiler passes are where you'd register multiple consumers or decorate the producer — if you're in Laravel, replicate this with a small registry keyed by consumer name rather than trying to emulate the bundle's container logic.