Elasticsearch Integration Laravel Package

Getting Started

Minimal Setup

1. Install & Configure

   composer require covertnija/elasticsearch-integration
   

   Update .env with your Elasticsearch hosts (CSV format):

   ELASTICSEARCH_HOSTS=127.0.0.1:9200,127.0.0.2:9200
   

2. First Use Case: Querying Elasticsearch Inject the client in a service/controller:

   use Covertnija\ElasticsearchIntegration\Client\ElasticsearchClientInterface;
   
   class ProductSearchController
   {
       public function __construct(
           private ElasticsearchClientInterface $elasticsearchClient
       ) {}
   
       public function search(): void
       {
           $response = $this->elasticsearchClient->search([
               'index' => 'products',
               'body'  => ['query' => ['match_all' => new \stdClass()]]
           ]);
           // Process $response
       }
   }
   

3. Verify Configuration Check config/packages/elasticsearch_integration.yaml for defaults and override as needed.

Implementation Patterns

1. Service Integration

• Autowiring: Use ElasticsearchClientInterface in any service/controller.
• Lazy Initialization: The client is lazy-loaded during first use, avoiding connection overhead during cache:clear.

2. Load Balancing & Failover

• Round-Robin: Automatically distributes requests across configured hosts.
• Failover: Failed nodes are skipped; the next node in the list is used.
• Dynamic Hosts: Update ELASTICSEARCH_HOSTS in .env to add/remove nodes without code changes.

3. Logging & Observability

• Kibana-Compatible Logs: Logs include @timestamp for direct ingestion into Kibana.
• Monolog Handler: Enable/disable via config:

  # config/packages/elasticsearch_integration.yaml
  elasticsearch_integration:
      logging:
          enabled: true
          level: debug
  

4. Configuration Overrides

• Environment Variables: Use %env(int:...%)% or %env(csv:...%)% for dynamic values.
• Programmatic Setup: Override defaults in a compiler pass or service:

  $this->container->setParameter('elasticsearch_integration.hosts', ['custom:host:9200']);
  

5. Bulk Operations

• Bulk API: Use the bulk() method for efficient indexing:

  $this->elasticsearchClient->bulk([
      'index' => 'products',
      'body'  => [
          ['index' => ['_id' => 1]],
          ['price' => 19.99],
          ['index' => ['_id' => 2]],
          ['price' => 29.99],
      ]
  ]);
  

6. Testing

• Mocking: Use ElasticsearchClientInterface in tests with a mock client.
• Integration Tests: Spin up a local Elasticsearch instance (e.g., Docker) and test against it.

Gotchas and Tips

Pitfalls

1. Connection Pooling:

   • The client reuses HTTP connections under the hood, but avoid long-running scripts that hold connections open unnecessarily.
   • For CLI commands, ensure connections are closed explicitly if needed:

     $this->elasticsearchClient->getConnection()->close();
     

2. Host Format:

   • Invalid Hosts: Ensure ELASTICSEARCH_HOSTS uses host:port format (e.g., 127.0.0.1:9200). Malformed entries will be skipped with a warning.

3. Logging Overhead:

   • Disable Logging in Production: Set logging.enabled: false in config to reduce overhead if logs aren’t needed.

4. Symfony Cache:

   • Lazy Loading: The client is lazy-loaded, but avoid circular dependencies that trigger early initialization (e.g., in onKernelRequest).

5. Elasticsearch Version:

   • Compatibility: The package targets Elasticsearch 8.x+. Avoid mixing with older versions (e.g., 7.x), as APIs may differ.

Debugging Tips

1. Enable Debug Logging:

   elasticsearch_integration:
       logging:
           level: debug
   

   Check logs for connection attempts, failovers, and query details.

2. Check Host Normalization:

   • If hosts aren’t working, verify .env or config:

     php bin/console debug:container elasticsearch_integration.client
     

     Look for the hosts parameter in the output.

3. Failover Debugging:

   • Use ELASTICSEARCH_HOSTS=badhost:9200,goodhost:9200 to test failover behavior.

Extension Points

1. Custom Client Configuration:

   • Extend the client by binding your own implementation of ElasticsearchClientInterface:

     services:
         App\Service\CustomElasticsearchClient:
             decorates: 'elasticsearch_integration.client'
     

2. Add Middleware:

   • Decorate the client to add request/response middleware:

     $client->getConnection()->addMiddleware(function (Request $request, callable $next) {
         $request->setHeader('X-Custom-Header', 'value');
         return $next($request);
     });
     

3. Custom Monolog Formatter:

   • Override the default formatter by configuring a custom handler:

     monolog:
         handlers:
             elasticsearch:
                 type: service
                 id: App\Logger\CustomElasticsearchHandler
     

4. Health Checks:

   • Integrate with Symfony’s health system by adding a check:

     use Symfony\Component\HealthCheck\HealthCheckInterface;
     use Symfony\Component\HealthCheck\HealthResult;
     
     class ElasticsearchHealthCheck implements HealthCheckInterface
     {
         public function check(HealthCheck $healthCheck): HealthResult
         {
             return $this->elasticsearchClient->ping()
                 ? HealthResult::healthy()
                 : HealthResult::unhealthy('Elasticsearch unavailable');
         }
     }
     

     Register it in config/services.yaml:

     services:
         Symfony\Component\HealthCheck\HealthCheck\ElasticsearchHealthCheck:
             arguments:
                 $elasticsearchClient: '@elasticsearch_integration.client'
             tags: ['health_check.check']
     

Performance Tips

1. Bulk Indexing:

   • Use the bulk() method for batch operations to minimize HTTP overhead.

2. Connection Reuse:

   • The client reuses connections, but limit concurrent requests to avoid overwhelming Elasticsearch.

3. Environment-Specific Config:

   • Use %kernel.environment% to switch configs:

     elasticsearch_integration:
         hosts: '%env(ELASTICSEARCH_HOSTS)%'
         logging:
             enabled: '%kernel.debug%'
     

4. Avoid Blocking Calls:

   • Offload long-running Elasticsearch operations to a queue (e.g., Symfony Messenger) to prevent timeouts.