Laravel Kafka Logger Laravel Package

Getting Started

Prerequisites — this package is only a thin Laravel logging driver around ext-rdkafka. Before touching composer, make sure the PHP extension is present:

pecl install rdkafka
# or: apt-get install php-rdkafka / docker-php-ext-enable rdkafka
php -m | grep rdkafka

Without the extension the driver will fail at boot (class RdKafka\Producer not found), so this is the first thing to verify.

Install and register — with Laravel auto-discovery the service provider is registered for you:

composer require hhxsv5/laravel-kafka-logger

If discovery is disabled, add Hhxsv5\LaravelKafkaLogger\KafkaLoggerServiceProvider (or the equivalent class in the repo's src/) to config/app.php.

Add a channel to config/logging.php using the kafka driver:

'channels' => [
    'kafka' => [
        'driver'  => 'kafka',
        'brokers' => env('KAFKA_BROKERS', '127.0.0.1:9092'),
        'topic'   => env('KAFKA_LOG_TOPIC', 'laravel-logs'),
        'level'   => env('LOG_LEVEL', 'debug'),
    ],
],

Matching .env:

KAFKA_BROKERS=kafka-1:9092,kafka-2:9092
KAFKA_LOG_TOPIC=app-logs-prod

First use case — write one message and confirm it lands in the topic before wiring it into anything else:

Log::channel('kafka')->info('kafka logger smoke test', ['service' => 'billing']);

Then consume it back with kafka-console-consumer --topic app-logs-prod --from-beginning (or your platform's UI). If that round-trip works, you understand the entire surface area of the package — everything else is Laravel's logging stack.

Implementation Patterns

Explicit channel, never the default. Keep stack/single/daily as the default and address Kafka deliberately:

Log::channel('kafka')->warning('payment retry', [
    'order_id' => $order->id,
    'attempt'  => $attempt,
]);

Promoting Kafka to the default channel makes every framework-level log line a network call and turns a broker blip into an application-wide stall. Reserve that for dedicated log-aggregation services.

Fan-out with a stack. The common day-to-day setup is: keep local files for on-box debugging, and mirror to Kafka for centralised search:

'stack' => [
    'driver'   => 'stack',
    'channels' => ['daily', 'kafka'],
],

Use this for the default so existing Log::info() calls keep working unchanged.

Correlation IDs via middleware. Kafka logs are far more useful when every line carries a request identity. Inject context once instead of threading it through every call:

// app/Http/Middleware/LogContext.php
public function handle($request, Closure $next)
{
    Log::withContext([
        'request_id' => (string) Str::uuid(),
        'user_id'    => optional($request->user())->id,
        'path'       => $request->path(),
    ]);

    return $next($request);
}

Because the driver sits behind Monolog, withContext applies to the Kafka channel as well — no package-specific work needed.

Structured context, not string interpolation. Consumers downstream (ELK, Datadog, ClickHouse) want parseable fields. Prefer:

Log::channel('kafka')->info('order.created', ['order_id' => $id, 'total' => $total]);

over "order {$id} created for {$total}". Ensure context values are JSON-serialisable — objects and resources will either be dropped or blow up depending on the formatter.

Topic-per-concern. Rather than one firehose topic, configure multiple channels pointing at different topics so consumers can subscribe narrowly:

'kafka_audit' => [
    'driver'  => 'kafka',
    'brokers' => env('KAFKA_BROKERS'),
    'topic'   => 'audit-events',
    'level'   => 'info',
],
'kafka_errors' => [
    'driver'  => 'kafka',
    'brokers' => env('KAFKA_BROKERS'),
    'topic'   => 'app-errors',
    'level'   => 'error',
],

Channel names are free; the topic is what costs money and disk in the cluster, so name topics with environment and service baked in (prod-billing-errors).

Queue workers and jobs. Log from inside jobs with the job's identifiers in context (job_id, attempt, queue). This is the highest-value place for Kafka logging because worker output is otherwise scattered across N machines. Remember queues already serialise the job — keep context lean.

Sample high-volume debug paths. If a hot loop writes to a Kafka channel, the network round-trip dominates. Gate it:

if (random_int(1, 100) === 1) {
    Log::channel('kafka')->debug('loop state', $state);
}

Gotchas and Tips

The package is stale and small. Last release mid-2020 and ten stars means: expect rough edges on newer Laravel majors, and budget for a fork. Check that the service provider's extend() call still matches the logging manager signature in your version before assuming the driver resolves. If it doesn't, the fix is usually a one-line signature change in a forked copy — pin your fork in composer.json with a VCS repository entry.

Synchronous publishing is the main operational risk. A Kafka log call does real network I/O. If brokers are slow or unreachable, your request slows or blocks — and a stack channel does not fall back to single on failure; it just throws. Mitigations:

• Keep Kafka off the default channel for latency-sensitive routes.
• Wrap risky writes in try { ... } catch (\Throwable $e) { /* file log */ }.
• Tune producer settings (linger.ms, queue.buffering.max.ms, message.timeout.ms) if the package exposes them; otherwise configure via a custom handler.

Never log Kafka failures to Kafka. A naïve try/catch that re-logs the exception to the same channel produces a retry storm or an infinite recursion. Catch and write to a file channel or stderr.

Message size limit. Kafka's default message.max.bytes is 1 MB. Logging a full request/response payload will silently produce MSG_SIZE_TOO_LARGE per-message errors from librdkafka that the package will not surface. Truncate large context and keep total payloads small.

Debugging checklist, in order:

1. php -m | grep rdkafka — extension loaded?
2. php artisan tinker then Log::channel('kafka')->info('x') — does the driver resolve?
3. Broker reachability: kafka-broker-api-versions.sh --bootstrap-server ... from the app host (not your laptop — VPC/firewall differences are the usual culprit).
4. Check librdkafka's own errors: they go to PHP's error log, not Laravel's channels, when the driver is misconfigured.
5. php artisan config:cache after .env changes — a cached config with old brokers is a classic "it works in tinker, not in prod".

Config caching trap. Any env-driven broker/topic lookup must live in config/logging.php, never inline in a service provider or controller, or config:cache will freeze a stale value.

Formatter behaviour differs from file channels. File logs default to the Monolog LineFormatter; the Kafka driver typically JSON-serialises. Don't assume a shared parser will handle both — give Kafka consumers a JSON schema up front and keep field names stable.

Extension points to reach for when the package is too thin:

• A custom Monolog handler (extend the package's handler or write your own RdKafka\Producer handler) registered via Log::extend() in a service provider — this is the cleanest way to add buffering, batch flushing, or custom serialisation without forking.
• A custom Formatter injected into the channel if you need Avro/Protobuf instead of JSON.
• If you outgrow the package entirely, swapping to mateusjunges/laravel-kafka or a dedicated Kafka client in an event listener is a small, well-bounded refactor — the call sites are just Log::channel(...).