Deduplication Laravel Package

Getting Started

• composer require rabbitevents/deduplication — this pulls in rabbitevents/listener ^10.0, so confirm you are already on the 10.x listener before installing; on 9.x you get the package but no deduplication, because 9.x-published messages carry no ID.
• First place to look is config/cache.php. Pick an existing store that every worker of the queue can reach (redis, database, or memcached) and point the package at it with RABBITEVENTS_DEDUPLICATION_STORE. Nothing else to configure — no RabbitMQ plugin, no migration, no queue declaration changes.
• Second place to look is your worker/deploy config: RABBITEVENTS_DEDUPLICATION_STORE must be present in the environment of every consumer process, not just the web app. Set it in .env, in the supervisor/ECS/K8s env for consumers, and in config/ if you resolve it at config-cache time.
• First use case worth proving out: a listener whose ack can be lost — connection drop, broker restart, worker killed after the handler returned. Before this package, that redelivery re-ran the listener. Pick one such listener in staging, kill the worker mid-ack, and confirm the second delivery is skipped.
• Keep the mental model straight from day one: this package closes the "listeners succeeded but ack was lost" window. It does not close the "worker died while listeners were running" window — those messages are still handled again, so idempotency in listeners remains your responsibility.

Implementation Patterns

• Treat deduplication as a delivery-layer concern, not a business-logic one. Wire it in once at the worker level and leave listener code untouched. The package is only useful if it is uniformly on for a queue's consumers; a partially-configured fleet is worse than none, because behaviour varies by which worker picks up the message.
• Pair it with an explicit idempotency strategy for mid-flight death. The documented gap is a worker dying while listeners run. The clean pattern is: cheap dedup from this package for the lost-ack case, plus an idempotency key / upsert / "already processed" check inside listeners with real side effects (payments, emails, webhooks).
• Per-queue granularity is a feature, use it deliberately. IDs are tracked per queue, so a message fanned out to several queues is handled once in each — which is what you want. Do not try to build cross-queue "handle exactly once globally" semantics on top of it; if a message legitimately needs handling in three queues, it will be handled three times.
• Standardise the store per environment, not per service. All workers of a queue must share one store. Redis is the usual choice: low latency on the hot path, natural TTLs, and already present in most stacks. Use database only if you accept a write per message in your primary DB and have the cache table migrated.
• Roll out store-first, package-second. Provision/verify the shared store and its connectivity from every consumer host before deploying the package. A missing store entry stops the worker at start, which turns a cache-config typo into a queue outage.
• Make the degradation path observable. An unreachable store means messages are handled without deduplication and a warning is logged. Route those warnings into your alerting (a log-based metric or Sentry grouping) so a silent Redis outage doesn't silently revert you to pre-package behaviour while the queue keeps draining and looks healthy.
• Watch the store's footprint like any other cache. Keys accumulate per handled message per queue. Use a store with eviction/TTL policies you understand, monitor memory, and keep dedup traffic off a Redis instance that also serves latency-critical app cache if you can afford a separate logical DB.
• In tests, point the store at a store isolated per test process (e.g. a dedicated Redis DB or the array store) so assertions about "handled twice" are deterministic and don't leak between tests. Assert both directions: a redelivered-after-success message is skipped, and a redelivered-after-failure message still runs.

Gotchas and Tips

• A store missing from config/cache.php stops the worker at start. This is the loudest failure mode and the one most likely to bite on first deploy — commonly because RABBITEVENTS_DEDUPLICATION_STORE names a connection that exists in .env but not in a config-cached config/cache.php, or names a database store whose migration never ran. Check php artisan config:cache behaviour in the deploy pipeline after setting the variable.
• An unreachable store fails open, not closed. Messages are handled without deduplication and only a warning is logged. That is the right default for availability, but it means an outage of your Redis/database/memcached silently removes the protection this package exists to provide. Alert on the warning; don't rely on reading logs after an incident.
• Never point this at a per-host store. file, array, or any node-local cache gives each worker its own view, so deduplication only works when the same worker happens to get the redelivery. The config will not stop you; the release notes only require "shared".
• Mid-flight worker death is explicitly not covered. A worker killed while listeners were running produces a redelivery that will run again. Idempotent listeners are still mandatory; treat this package as reducing a common class of duplicates, not as exactly-once delivery. RabbitMQ at-least-once semantics are unchanged.
• 9.x-published messages have no ID and are skipped by deduplication. During a mixed-version rollout — or for anything still publishing via the 9.x listener — some messages are deduplicated and some aren't. Full coverage requires publishers on 10.x; plan the upgrade of producers, not just consumers.
• Per-queue IDs mean duplicate side effects across queues are expected. A message routed to orders and analytics is handled once in each queue. If your listener writes to a shared resource from both queues, that is a business-level duplicate and deduplication won't hide it.
• Treat the cache store as a dependency with a TTL policy. If the store evicts keys aggressively (memory pressure, allkeys-lru, short TTLs), a late redelivery may no longer find its marker and will be handled again. Size the store for your redelivery window and avoid mixing it with volatile caches.
• Extension points are intentionally thin — the store selection (env var) is the only knob. If you need per-queue stores, custom key schemes, or metrics, that's application-side work (log parsing, a custom cache store, or a wrapper listener) rather than configuration. Keep an eye on the package's config surface as it matures; v1.0.0 is a single-behaviour release.
• Debugging recipe: if you suspect deduplication isn't working, check in order — is the store configured and reachable from that host, was the message published by a 10.x listener (i.e. does it carry an ID), is the redelivery on the same queue, and did the original run actually complete all listeners successfully. If all four hold, it should be skipped; if it isn't, capture the message ID and inspect the store keys directly.