Data Watcher Bundle Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require benjamin-rqt/data-watcher-bundle
   

   Ensure BenjaminRqt\DataWatcherBundle\DataWatcherBundle is registered in config/bundles.php.

2. Basic Configuration:

   # config/packages/data_watcher.yaml
   data_watcher:
       from_email: 'monitoring@yourdomain.com'
       recipients:
           - 'devops@yourdomain.com'
       app_name: 'YourApp'
   

3. First Use Case: Define a watch rule in a Doctrine entity to monitor a field for anomalies (e.g., sudden spikes in user_visits):

   // src/Entity/UserVisit.php
   use BenjaminRqt\DataWatcherBundle\Annotation\Watch;
   
   #[Watch(
       field: 'visits',
       threshold: 1000, // Trigger if >1000 visits in a row
       window: 'PT1H',   // Check every hour
       description: 'High traffic spike detected'
   )]
   class UserVisit { ... }
   

4. Trigger a Test Check: Run the scheduler manually to test:

   php bin/console messenger:consume data-watcher -vv
   

Implementation Patterns

Core Workflow

1. Define Rules: Use the @Watch annotation on Doctrine entities to specify fields, thresholds, and time windows.

   #[Watch(
       field: 'order_total',
       threshold: 10000,
       window: 'PT1D', // Daily check
       operator: '>',  // Supports: >, <, !=
       notify_on_recovery: true
   )]
   

2. Configure Scheduling: Set up a cron job (via Symfony Scheduler or cron-expression) to run checks periodically:

   # config/packages/scheduler.yaml
   scheduler:
       jobs:
           data_watcher_check:
               type: 'doctrine'
               schedule: '0 0 * * *' # Daily at midnight
               class: 'BenjaminRqt\DataWatcherBundle\Command\CheckAnomaliesCommand'
   

3. Customize Notifications: Override the default Twig template (templates/data_watcher/email.html.twig) or extend the DataWatcherEvent to add custom logic:

   // src/EventListener/DataWatcherListener.php
   use BenjaminRqt\DataWatcherBundle\Event\DataWatcherEvent;
   
   class DataWatcherListener implements EventSubscriberInterface {
       public static function getSubscribedEvents(): array {
           return [
               DataWatcherEvent::ANOMALY_DETECTED => 'onAnomaly',
           ];
       }
   
       public function onAnomaly(DataWatcherEvent $event): void {
           if ($event->getEntity() instanceof UserVisit) {
               // Add Slack notification or other integrations
           }
       }
   }
   

4. Query Optimization: For large datasets, use DQL hints or native queries to optimize anomaly detection:

   // src/Repository/UserVisitRepository.php
   public function findAnomalies(int $threshold, \DateTimeInterface $windowStart): array {
       return $this->createQueryBuilder('uv')
           ->where('uv.visits > :threshold')
           ->andWhere('uv.created_at BETWEEN :start AND :end')
           ->setParameter('threshold', $threshold)
           ->setParameter('start', $windowStart)
           ->setParameter('end', (clone $windowStart)->modify('+1 hour'))
           ->getQuery()
           ->getResult();
   }
   

Integration Tips

• Symfony Messenger: Use the data-watcher transport to queue checks for async processing:

  # config/packages/messenger.yaml
  framework:
      messenger:
          transports:
              data-watcher: '%env(MESSENGER_TRANSPORT_DSN)%'
          routing:
              'BenjaminRqt\DataWatcherBundle\Message\CheckAnomaliesMessage': data-watcher
  

• Doctrine Events: Subscribe to postPersist/postUpdate to trigger real-time checks for critical fields:

  // src/EventSubscriber/RealTimeWatchSubscriber.php
  use Doctrine\Common\EventSubscriber;
  use Doctrine\ORM\Event\OnFlushEventArgs;
  
  class RealTimeWatchSubscriber implements EventSubscriber {
      public function getSubscribedEvents(): array {
          return ['onFlush'];
      }
  
      public function onFlush(OnFlushEventArgs $args): void {
          $em = $args->getEntityManager();
          foreach ($em->getUnitOfWork()->getScheduledEntityInsertions() as $entity) {
              if ($entity instanceof UserVisit && $entity->getVisits() > 5000) {
                  // Dispatch a real-time check message
              }
          }
      }
  }
  

• API Endpoints: Expose a /_health/data-watcher endpoint to verify the bundle is running:

  // src/Controller/DataWatcherController.php
  use Symfony\Component\HttpFoundation\Response;
  use Symfony\Component\Routing\Annotation\Route;
  
  #[Route('/_health/data-watcher', name: 'data_watcher_health')]
  public function healthCheck(): Response {
      return new Response(
          json_encode(['status' => 'ok', 'last_check' => now()->format('Y-m-d H:i:s')]),
          200,
          ['Content-Type' => 'application/json']
      );
  }
  

Gotchas and Tips

Pitfalls

1. Cron Expression Parsing:

   • Ensure dragonmantank/cron-expression is installed (required for scheduler triggers).
   • Validate cron syntax in scheduler.yaml; invalid expressions will silently fail.
   • Fix: Use crontab.guru to test expressions before deployment.

2. Threshold Sensitivity:

   • Hardcoded thresholds (e.g., threshold: 1000) may trigger false positives/negatives.
   • Solution: Use dynamic thresholds based on historical data:

     #[Watch(
         field: 'user_registrations',
         threshold: 'PT1H', // Dynamic: "10x the hourly average"
         dynamic_threshold: true
     )]
     

3. Email Delays:

   • If emails are delayed, check the messenger:consume queue for stuck messages.
   • Debug: Run php bin/console messenger:failed:list and retry failed messages.

4. Entity Changes:

   • Adding/removing @Watch annotations requires clearing the cache:

     php bin/console cache:clear
     

5. Time Window Misalignment:

   • window: 'PT1H' checks data in the past hour. Ensure your created_at timestamps are accurate.
   • Tip: Use UTC for consistency across environments.

Debugging

1. Log Anomalies: Enable debug mode to log anomaly details:

   # config/packages/data_watcher.yaml
   data_watcher:
       debug: true
   

   Check logs at var/log/dev.log for raw anomaly data.

2. Dry Run Mode: Test rules without sending emails by setting:

   data_watcher:
       dry_run: true
   

3. Query Profiling: Use Doctrine’s query profiler to optimize slow checks:

   php bin/console debug:query --format=json > queries.json
   

Extension Points

1. Custom Anomaly Types: Extend the Anomaly class to support new detection logic (e.g., machine learning):

   // src/Entity/CustomAnomaly.php
   use BenjaminRqt\DataWatcherBundle\Entity\Anomaly;
   
   class CustomAnomaly extends Anomaly {
       public function isAnomaly(): bool {
           // Custom logic (e.g., statistical outliers)
           return $this->getValue() > $this->getExpectedValue() * 1.5;
       }
   }
   

2. Webhook Notifications: Add a WebhookNotifier to send alerts to third-party services:

   // src/Notifier/WebhookNotifier.php
   use BenjaminRqt\DataWatcherBundle\Notifier\NotifierInterface;
   
   class WebhookNotifier implements NotifierInterface {
       public function notify(Anomaly $anomaly): void {
           $client = new \GuzzleHttp\Client();
           $client->post('https://your-webhook-url', [
               'json' => $anomaly->toArray(),
           ]);
       }
   }
   

   Register it in data_watcher.yaml:

   data_watcher:
       notifiers:
           - 'BenjaminRqt\DataWatcherBundle\Notifier\EmailNotifier'
           - 'App\Notifier\WebhookNotifier'
   

3. Dashboard Integration: Expose anomaly history via API for a custom dashboard:

   // src/Controller/AnomalyController.php
   use BenjaminRqt\DataWatcherBundle\Repository\AnomalyRepository;
   
   #[Route('/api/anomalies', name: 'api_anomalies')]