Kafka Bundle Laravel Package

Getting Started

Read this first: geekcell/kafka-bundle is a Symfony bundle, not a Laravel package. It registers itself through Symfony\Component\HttpKernel\Bundle\Bundle, ships a DependencyInjection/Extension and a YAML config tree, and is wired through Symfony's container. In Laravel it will not auto-discover, and none of its service definitions will resolve without a compatibility shim. Combined with the facts that it is archived (last release Jan 2023), has 2 stars and 0 dependents, you should treat it as a reference implementation of Kafka wiring rather than a dependency you install into a Laravel app. That said, the patterns it encodes are worth stealing.

If you are evaluating it inside a Symfony service (or a Laravel app with symfony/* components pulled in), the minimal loop is:

1. Check the hard requirements. You almost certainly need the native ext-rdkafka extension plus librdkafka on the host. Confirm in composer.json whether the bundle depends on ext-rdkafka directly or on a wrapper like enqueue/rdkafka. This is the single biggest install-time blocker and the reason most "painless integration" claims fall over in CI.
2. Install and register.

   composer require geekcell/kafka-bundle
   

   // config/bundles.php (Symfony)
   return [
       GeekCell\KafkaBundle\KafkaBundle::class => ['all' => true],
   ];
   

3. Configure brokers and consumer group. The full option list lives in src/DependencyInjection/Configuration.php — read that file before writing any YAML, it is the actual documentation. A typical shape:

   # config/packages/geekcell_kafka.yaml
   geekcell_kafka:
       brokers: '%env(KAFKA_BROKERS)%'      # e.g. "kafka:9092"
       group_id: '%env(KAFKA_GROUP_ID)%'
       # topics / serializer / security options per the Configuration tree
   

4. Where to look first, in order:
   • src/DependencyInjection/Configuration.php — exhaustive config surface.
   • src/Resources/config/services.yaml — the real service IDs. Autowiring in this bundle is opinionated; if a producer interface isn't aliased, you will need to bind it yourself.
   • src/Command/ — the consumer entrypoint (usually something like kafka:consume). This is what your supervisor units will call.
   • tests/ — the only trustworthy usage examples for an archived bundle.
5. First use case to prove it works: publish one JSON message to a topic from a controller, then consume it in a worker with a log-line handler. Do not start with handlers, serializers, or Avro. Prove broker connectivity and offset commits first; everything else is decoration on top of those two things.

For a Laravel codebase, the equivalent skeleton you should actually build (and which this bundle's structure can inform) is: a KafkaServiceProvider binding a singleton producer/consumer client from config/kafka.php, a php artisan kafka:consume command wrapping the same loop, and a Kafka facade for publishers. That is roughly 150 lines and gives you something maintained.

Implementation Patterns

Producer: inject, don't instantiate. The bundle's value proposition is that the producer is a container-managed service. In Symfony, autowire the producer interface into the service that emits events; never new a producer inside a domain object. The pattern that scales is a thin application-level publisher that owns serialization and topic naming, with the bundle's producer hidden behind it:

final class OrderEventPublisher
{
    public function __construct(private ProducerInterface $producer) {}

    public function orderPlaced(Order $order): void
    {
        $this->producer->produce('orders.v1', [
            'event'     => 'order.placed',
            'id'        => $order->id,
            'placed_at' => $order->placedAt->format(DATE_ATOM),
        ]);
    }
}

Consumer: long-running worker, one process per group. The bundle's consumer is a blocking poll() loop, not a request-scoped service. Run it under supervisor/systemd, cap memory (--memory-limit style), and make it restartable. In Laravel terms this is queue:work — same operational discipline, same "don't run it in a web request" rule.

Handler-per-topic via tagged services. The opinionated Symfony pattern here is a MessageHandlerInterface (or an attribute/kafka.handler tag) so the compiler pass collects handlers and routes by topic. Reproduce that in Laravel with a Kafka::handler('orders.v1', OrderPlacedHandler::class) registry rather than a switch ($topic) inside the command — the switch becomes unreadable the moment you have five topics.

Translate the DI idioms, don't fight them. Mapping table you'll need when moving between the two ecosystems:

Commit strategy is a design decision, not a config flag. Decide explicitly: commit after each handler succeeds (at-least-once, safe, slower) versus batch commit (faster, duplicate risk on crash). Write handlers to be idempotent — key by message ID, upsert, or dedupe table. This is the single pattern that determines whether the integration is boring or an incident generator.

Local development: Redpanda over Kafka. A single-node Redpanda container boots in seconds and speaks the Kafka protocol; pair it with kafka-ui or redpanda-console. You want the ability to rpk topic consume orders.v1 while debugging far more than you want a faithful three-broker cluster on a laptop.

Testing. Never let unit tests touch a broker. Wrap the producer in your own interface and bind a fake in tests that records calls; assert on (topic, payload) tuples. For integration tests, hit a Testcontainers/Redpanda instance and assert round-trip through the real consumer, but keep that suite separate.

Gotchas and Tips

• It is archived. There will be no fixes. Symfony 6.3+/7 container changes, PHP 8.3+ deprecations, and any librdkafka API drift will not be patched. Pin your PHP and Symfony versions, and budget for a fork if you adopt it.
• Laravel is not a supported target. Zero dependents means zero community-tested Laravel usage. Do not assume package discovery, config publishing, or facade support — none of it exists. If you need Kafka in Laravel, enqueue/rdkafka, mateusjunges/laravel-kafka, or a hand-rolled ext-rdkafka wrapper are the realistic options.
• ext-rdkafka is a native extension. It is not a composer package you can composer require away. Docker images need librdkafka-dev at build time; Alpine and Debian need different incantations; PHP version and librdkafka version must be compatible. This is where 80% of setup time disappears.
• Consumer group rebalancing surprises. Adding a second consumer instance triggers a rebalance and a pause in consumption. Any handler longer than max.poll.interval.ms gets the consumer kicked out of the group and the message redelivered — leading to duplicate processing and confusing "why did this run twice" investigations. Keep handlers short; push slow work to a queue.
• Auto-commit is a trap. If the bundle defaults to enable.auto.commit=true, offsets commit on a timer regardless of handler success, so a crash silently drops messages. Verify the default in the config tree and switch to manual commit-after-success.
• Silent config errors via config:cache. Symfony's container compilation will happily bake in stale geekcell_kafka values. After changing KAFKA_BROKERS, run bin/console cache:clear — a "connection refused to old-broker-host" error is almost always a cached container, not a broker problem.
• Service not found / cannot autowire is the second most common failure with opinionated bundles: the interface exists but has no alias. Register it yourself in services.yaml:

  GeekCell\KafkaBundle\Contract\ProducerInterface:
      alias: GeekCell\KafkaBundle\Producer\Producer
  

• No schema registry, no schema safety. Expect plain JSON. Once a producer ships a renamed field, old consumers break at runtime, not at deploy. Version your topics (orders.v1 → orders.v2) and keep producers backward-compatible by only adding fields.
• No built-in DLQ. Retry/DLQ is your responsibility. Pattern: catch handler exceptions, republish to <topic>.dlq with the original payload and error context, then commit. Without it, one poison message blocks the partition forever.
• Debugging toolkit. rpk topic consume <topic> --offset start, rpk group describe <group> to see lag and assigned partitions, and log the partition/offset/key on every consumed message. If your logs don't include offset, you cannot reconstruct what happened.
• Message keys control ordering. Kafka guarantees order only within a partition. If ordering matters, set the key deliberately (e.g. order_id); if it doesn't, accept that you'll get messages out of order and design handlers accordingly.
• Extension points worth knowing. The producer can be decorated (Symfony's decorates: / Laravel's extend) to add tracing, metrics, and structured logging without touching call sites. The serializer is usually swappable via config — make it explicit rather than relying on the default, so you control date formats and null handling.
• Health checks. A broker outage should surface in your /health endpoint, not as a 30-second hang in a user request. Wrap produce calls with a timeout and fail fast or buffer; blocking a web request on a Kafka acks=all write is a latency incident waiting to happen.