Elastica Fast Populate Bundle Laravel Package

Getting Started

1. Install both bundles. This is a companion to FOSElasticaBundle, not a replacement. You need FOSElasticaBundle already indexing data before this package does anything useful.

   composer require steamulo/elastica-fast-populate-bundle
   

2. Register the bundle in config/bundles.php (for Symfony 4+):

   STM\ElasticaFastPopulateBundle\STMElasticaFastPopulateBundle::class => ['all' => true],
   

3. Configure a real message transport. Because the whole point is distributing work to consumers, the bundle is only useful when messages leave the process. In practice that means a queued transport (AMQP/RabbitMQ, Redis, Doctrine, or Beanstalkd via Symfony Messenger), not sync. Look at the bundle's DependencyInjection/Configuration.php and its service definitions first — this is a low-star package, so the source is the documentation.

4. First use case: run the populate command the bundle exposes (typically a variant of fos:elastica:populate) and then start a pool of workers:

   php bin/console <populate-command> --index=product
   php bin/console messenger:consume <transport_name> --time-limit=3600
   

   With a handful of workers running concurrently, you should see wall-clock time drop compared to the single-threaded fos:elastica:populate, especially on indices with tens of thousands of documents or more.

Implementation Patterns

Fan-out populate with N workers. The core pattern is: producer slices the index into batches, each batch becomes a message, workers index those batches in parallel. Run several messenger:consume processes (Supervisor / systemd units) sized to your Elasticsearch cluster's write capacity, not to your CPU count.

Per-index/per-type invocation. Populate one index at a time rather than all indices in one command, so a failure is scoped and retries are cheap:

php bin/console <populate-command> --index=product --type=product

Wire into a deploy/migration step. A typical workflow: after schema changes or a fresh ES cluster spin-up, run the populate command in CI/deploy tooling, let workers drain the queue, then flip traffic. Pair it with FOSElasticaBundle's alias/reindex strategy so reads never go to a half-populated index.

Idempotent batch handlers. Treat each batch message as "index documents with IDs X..Y, replacing existing docs." On retry, the same message can safely re-index. This makes the queue tolerant of worker crashes.

Progress visibility. Since work is spread across workers, you lose the single-process progress bar. Track progress by logging per-batch counts (in your message handler or via a middleware) and compare against the source count (COUNT(*) on the ORM side) or against _cat/indices doc counts on the ES side.

Retry & poison-message handling. Configure the transport with a retry strategy (e.g. max_retries: 3, exponential backoff) and a failure transport. Individual batches failing on a single malformed row is the most common failure mode; you want those isolated to one message, not the whole populate.

Gotchas and Tips

• sync transport defeats the purpose. If the Messenger transport is sync, you've replaced one slow command with one equally slow command plus dispatch overhead. Verify the transport in config/packages/messenger.yaml before benchmarking.
• Elasticsearch refresh interval. The single biggest perf lever during a bulk populate is refresh_interval: -1 (or a large value) on the target index, then a refresh when done. This bundle doesn't magically do that for you; enable it manually or in your populate orchestration.
• Consumer memory creep. Elasticsearch PHP clients and Doctrine entity managers leak over time. Always run consumers with --time-limit or --memory-limit, and use --no-debug to avoid the profiler holding references.
• Version skew between producers and consumers. The producer command and the workers must run the same release of the bundle, FOSElasticaBundle, and your indexing code. Roll workers together with the app; don't deploy the app and leave stale consumers on the queue.
• Last release mid-2023, zero stars, zero dependents. Expect sparse documentation and no community answers. Read the bundle's Command/ and Message//MessageHandler/ directories directly. Pin the version in composer.json and be prepared to fork if you hit a Symfony 7 compatibility issue.
• Batch size is the tuning knob. Too small → transport overhead dominates; too large → workers don't balance and a single slow batch blocks the finish line. Start around 500–2000 docs per message and measure.
• Alias/zero-downtime caution. If you populate a live index in place, searches see partial data during the run. Populate into a new index and swap an alias, or accept a maintenance window.
• Debugging stuck jobs. If workers idle while the command seems to hang, check the transport binding/routing_key, that the consumer is listening on the right transport, and whether the producer is throwing on serialization (a common issue when messages carry entity references instead of IDs).
• Extension points. If the bundle doesn't expose the batching granularity you need, look for a configurable batch size under its stm_elastica_fast_populate (or similarly named) config key; failing that, decorate FOSElasticaBundle's provider/indexer service or override the message handler to control slicing.