Sync Bundle Laravel Package

Getting Started

Minimal Steps

1. Installation

   composer require nti/sync-bundle "dev-master"
   

   Ensure your project uses Symfony 2.x/3.x (last release was 2021).

2. Register the Bundle Add to AppKernel.php:

   new NTI\SyncBundle\NTISyncBundle(),
   

3. Define a Syncable Entity Annotate an entity with @NTI\SyncEntity and implement SyncRepositoryInterface:

   use Doctrine\ORM\Mapping as ORM;
   use NTI\SyncBundle\Annotation\NTI\SyncEntity;
   
   /**
    * @SyncEntity
    * @ORM\Entity(repositoryClass="AppBundle\Repository\MyEntitySyncRepository")
    */
   class MyEntity {
       // ...
   }
   

4. Configure Sync Mapping Define a SyncMapping for your entity (e.g., in a service or config):

   # config.yml
   nti_sync:
       mappings:
           my_entity:
               class: AppBundle\Entity\MyEntity
               identifier: id
               fields: [name, description]
   

5. Update Schema & Routes

   php app/console doctrine:schema:update --force
   

   Add to routing.yml:

   nti_sync:
       resource: "@NTISyncBundle/Resources/config/routing.yml"
   

6. Initialize Sync State Run the provided SQL snippet to populate nti_sync_state for each mapping.

7. First Sync Trigger Call the sync endpoint (e.g., via CLI or HTTP):

   php app/console nti:sync:run --mapping=my_entity
   

First Use Case: Syncing a Product Catalog

• Scenario: Sync products from an external API to your database.
• Steps:
  1. Annotate Product entity with @SyncEntity.
  2. Implement SyncRepositoryInterface to handle sync logic (e.g., fetch from API, merge changes).
  3. Configure SyncMapping for Product in config.yml.
  4. Trigger sync via CLI or HTTP endpoint (/sync/run?mapping=product).

Implementation Patterns

Core Workflow: Syncing Entities

1. Annotation-Driven Setup

   • Use @SyncEntity on entities to auto-register them for sync.
   • Example:

     /**
      * @SyncEntity
      * @ORM\Entity
      */
     class User { ... }
     

2. Repository Integration

   • Implement SyncRepositoryInterface to define sync behavior:

     class UserSyncRepository extends ServiceEntityRepository implements SyncRepositoryInterface {
         public function findForSync(SyncMapping $mapping) {
             // Fetch unsynced or changed records.
         }
         public function sync(SyncMapping $mapping, $data) {
             // Merge data into entity.
         }
     }
     

3. Parent-Child Sync

   • Use @SyncParent on ManyToOne fields to update parent timestamps:

     /**
      * @ManyToOne(targetEntity="Category")
      * @SyncParent(getter="getCategory")
      */
     private $category;
     

4. Mapping Configuration

   • Define mappings in config.yml or via YAML/XML:

     nti_sync:
         mappings:
             user:
                 class: AppBundle\Entity\User
                 identifier: id
                 fields: [username, email, last_sync]
     

5. Triggering Syncs

   • CLI: php app/console nti:sync:run --mapping=user
   • HTTP: POST to /sync/run with mapping=user parameter.
   • Cron Jobs: Schedule periodic syncs (e.g., 0 3 * * * php app/console nti:sync:run --mapping=user).

Integration Tips

1. External API Sync

   • Use findForSync() to fetch only changed records (e.g., via last_updated field).
   • Example:

     public function findForSync(SyncMapping $mapping) {
         return $this->createQueryBuilder('u')
             ->where('u.lastUpdated > :lastSync')
             ->setParameter('lastSync', $mapping->getLastSyncTimestamp())
             ->getQuery()
             ->getResult();
     }
     

2. Conflict Resolution

   • Override sync() to handle conflicts (e.g., prioritize local DB or external source):

     public function sync(SyncMapping $mapping, $data) {
         $entity = $this->find($data['id']);
         if ($entity->getVersion() > $data['version']) {
             // Local version is newer; skip or merge.
             return;
         }
         // Apply changes.
     }
     

3. Bulk Sync Optimization

   • Use Doctrine batch operations in sync() to reduce queries:

     $em = $this->getEntityManager();
     $em->beginTransaction();
     foreach ($data as $item) {
         $entity = $this->find($item['id']);
         $entity->setName($item['name']);
         $this->save($entity, $em);
     }
     $em->commit();
     

4. Logging

   • Extend the bundle to log sync operations (e.g., using Monolog):

     public function sync(SyncMapping $mapping, $data) {
         $this->logger->info('Syncing entity', ['mapping' => $mapping->getName(), 'data' => $data]);
         // ...
     }
     

5. Testing

   • Mock SyncRepositoryInterface in tests:

     $repository = $this->createMock(SyncRepositoryInterface::class);
     $repository->method('findForSync')->willReturn([]);
     $this->entityManager->expects($this->any())
         ->method('getRepository')
         ->with('AppBundle:User')
         ->willReturn($repository);
     

Gotchas and Tips

Pitfalls

1. Missing @SyncEntity Annotation

   • Issue: Entities without @SyncEntity are ignored during sync.
   • Fix: Ensure all syncable entities are annotated. Use a custom compiler pass to validate this at runtime:

     public function process(ContainerBuilder $container) {
         $entities = $this->findAnnotatedEntities();
         foreach ($entities as $entity) {
             if (!class_exists($entity) || !in_array(SyncEntity::class, class_uses($entity))) {
                 throw new \RuntimeException("Entity $entity must be annotated with @SyncEntity");
             }
         }
     }
     

2. Uninitialized SyncState

   • Issue: Sync fails if nti_sync_state lacks entries for mappings.
   • Fix: Run the provided SQL snippet after all SyncMapping configurations are defined.

3. Circular Dependencies in Parent-Child Sync

   • Issue: @SyncParent on bidirectional relationships can cause infinite loops.
   • Fix: Use getter to explicitly define the parent field:

     @SyncParent(getter="getParentEntity") // Avoids ambiguity.
     

4. Repository Not Implemented

   • Issue: SyncRepositoryInterface not implemented for annotated entities.
   • Fix: Extend ServiceEntityRepository and implement the interface:

     class MyEntitySyncRepository extends ServiceEntityRepository implements SyncRepositoryInterface {
         // Implement required methods.
     }
     

5. Timestamp Drift

   • Issue: last_sync timestamps may desync if clocks drift between systems.
   • Fix: Use UTC timestamps and validate in findForSync():

     ->where('u.lastSync < :now')
     ->setParameter('now', new \DateTime('now', new \DateTimeZone('UTC')))
     

Debugging Tips

1. Enable SQL Logging Add to config.yml:

   doctrine:
       dbal:
           logging: true
           profiling: true
   

   Check logs for generated queries during sync.

2. Dump Sync Data Override findForSync() to log fetched data:

   public function findForSync(SyncMapping $mapping) {
       $data = $this->createQueryBuilder('e')->getQuery()->getResult();
       file_put_contents('sync_debug.log', print_r($data, true));
       return $data;
   }
   

3. Check Sync State Query nti_sync_state to verify timestamps:

   SELECT * FROM nti_sync_state WHERE mapping_id = (SELECT id FROM nti_sync_mapping WHERE name = 'user');
   

4. Validate Mappings Ensure SyncMapping configurations match entity fields:

   $mapping = $container->get('nti_sync.mapping.user');
   var_dump($mapping->getFields()); // Should match entity properties.
   

Extension Points

1. Custom Sync Strategies
   • Extend the bundle by creating a