Vdm Library Bundle Laravel Package

Getting Started

Minimal Setup

1. Install the Bundle

   composer require 3slab/vdm-library-bundle
   

   Add to config/bundles.php:

   return [
       // ...
       Vdm\LibraryBundle\VdmLibraryBundle::class => ['all' => true],
   ];
   

2. Configure Required Services Edit config/packages/vdm_library.yaml (auto-generated) to define:

   • vdm_collect (HTTP/FTP/SFTP sources)
   • vdm_store (Elasticsearch/Doctrine)
   • vdm_backbone (broker transport like RabbitMQ, AMQP, etc.)

3. First Use Case: HTTP Pull Consumer Define a message handler class (e.g., src/MessageHandler/ProcessHttpDataHandler.php):

   namespace App\MessageHandler;
   
   use Vdm\LibraryBundle\Message\CollectHttpMessage;
   use Symfony\Component\Messenger\Attribute\AsMessageHandler;
   
   #[AsMessageHandler]
   class ProcessHttpDataHandler
   {
       public function __invoke(CollectHttpMessage $message)
       {
           // Process $message->getData()
           return new ProduceToStoreMessage($processedData);
       }
   }
   

4. Trigger a Job Dispatch a message via CLI or controller:

   php bin/console messenger:consume vdm_collect_http -vv
   

Implementation Patterns

Core Workflows

1. HTTP Data Collection Pipeline

graph TD
    A[Trigger HTTP Pull] --> B[CollectHttpMessage]
    B --> C[ProcessHttpDataHandler]
    C --> D[ProduceToStoreMessage]
    D --> E[Elasticsearch/Doctrine Store]

• Pattern: Chain CollectHttpMessage → Custom Handler → ProduceToStoreMessage.
• Retry Logic: Configure retries in vdm_collect.http:

  vdm_collect:
      http:
          sources:
              my_source:
                  url: "https://api.example.com/data"
                  retry:
                      max_attempts: 3
                      delay: 1000
  

2. Broker-Based Message Handling

• Producer Pattern (e.g., after processing):

  $message = new ProduceToStoreMessage($data);
  $this->messageBus->dispatch($message);
  

• Consumer Pattern (via Symfony Messenger):

  # config/packages/messenger.yaml
  framework:
      messenger:
          transports:
              vdm_backbone: '%env(VDM_BROKER_DSN)%'
          routing:
              'App\Message\ProduceToStoreMessage': vdm_backbone
  

3. Monitoring Integration

• Metrics Collection: Use Symfony’s Stopwatch or integrate with Prometheus:

  use Symfony\Component\Stopwatch\Stopwatch;
  
  class ProcessHttpDataHandler {
      public function __invoke(CollectHttpMessage $message, Stopwatch $stopwatch) {
          $event = $stopwatch->start('http_processing');
          // ... processing logic ...
          $event->stop();
      }
  }
  

Integration Tips

Doctrine Store

• Entity Mapping: Define a StoreEntity with @Vdm\Store\Annotation\Indexed for Elasticsearch sync:

  #[ORM\Entity]
  #[Vdm\Store\Annotation\Indexed(indexName: 'my_index')]
  class Product {
      // ...
  }
  

FTP/SFTP Sources

• Configure in vdm_collect.ftp:

  vdm_collect:
      ftp:
          sources:
              remote_files:
                  host: "ftp.example.com"
                  username: "%env(FTP_USER)%"
                  password: "%env(FTP_PASS)%"
                  directory: "/incoming"
                  file_pattern: "*.csv"
  

Custom Message Handling

• Extend base messages:

  namespace App\Message;
  
  use Vdm\LibraryBundle\Message\BaseMessage;
  
  class CustomCollectMessage extends BaseMessage {
      public function __construct(string $source, array $options) {
          parent::__construct($source, $options);
      }
  }
  

Gotchas and Tips

Pitfalls

1. Broker Connection Issues

   • Symptom: Messages fail silently or time out.
   • Fix: Verify VDM_BROKER_DSN in .env and check broker health:

     php bin/console debug:container vdm.backbone.producer
     

2. Elasticsearch Indexing Delays

   • Symptom: Data appears in Doctrine but not Elasticsearch.
   • Fix: Ensure vdm_store.elasticsearch.sync is enabled and the StoreEntity has the @Indexed annotation.

3. HTTP Retry Overload

   • Symptom: Retries cause excessive load on external APIs.
   • Fix: Limit retries and add exponential backoff:

     retry:
         max_attempts: 3
         delay: 1000
         multiplier: 2  # Exponential backoff
     

Debugging

• Enable Messenger Debugging:

  php bin/console messenger:failed:list
  php bin/console messenger:failed:remove ALL
  

• Log Message Flow:

  # config/packages/monolog.yaml
  monolog:
      handlers:
          messenger:
              type: stream
              path: "%kernel.logs_dir%/messenger.log"
              level: debug
  

Extension Points

1. Custom Metrics

   • Extend Vdm\LibraryBundle\Monitoring\MetricCollector to add custom metrics:

     namespace App\Monitoring;
     
     use Vdm\LibraryBundle\Monitoring\MetricCollector;
     
     class CustomMetricCollector extends MetricCollector {
         public function collectCustomMetric(string $name, $value) {
             $this->metrics[$name] = $value;
         }
     }
     

2. Message Transformers

   • Use Symfony Messenger’s MessageBus middleware to transform messages:

     use Symfony\Component\Messenger\Middleware\HandleMessage;
     
     class TransformMessageMiddleware {
         public function __invoke(HandleMessage $handle, $message) {
             if ($message instanceof CollectHttpMessage) {
                 $message->setData($this->transform($message->getData()));
             }
             return $handle($message);
         }
     }
     

3. Event Listeners

   • Listen to vdm.library.message.processed events:

     namespace App\EventListener;
     
     use Vdm\LibraryBundle\Event\MessageProcessedEvent;
     use Symfony\Component\EventDispatcher\Attribute\AsEventListener;
     
     #[AsEventListener(event: 'vdm.library.message.processed')]
     class LogProcessedMessageListener {
         public function __invoke(MessageProcessedEvent $event) {
             // Log or notify on message processing
         }
     }
     

Configuration Quirks

• Default Transport: If no vdm_backbone.transport is set, the bundle defaults to symfony://async, which may not suit high-throughput scenarios.
• Elasticsearch Sync: Ensure vdm_store.elasticsearch.client is properly configured (e.g., elasticsearch://user:pass@localhost:9200).
• FTP Passive Mode: For SFTP/FTPS, set passive: true in the config to avoid firewall issues:

  ftp:
      sources:
          remote_files:
              passive: true