Datamigration Bundle Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require appventus/datamigration-bundle
   

   Register the bundle in config/bundles.php:

   AppVentus\DataMigrationBundle\AppVentusDataMigrationBundle::class => ['all' => true],
   

2. Enable Tracking: Add the DataMigrationListener to your Doctrine event manager in config/packages/doctrine.yaml:

   doctrine:
       orm:
           event_listeners:
               AppVentus\DataMigrationBundle\EventListener\DataMigrationListener: ~
   

3. First Use Case:

   • Perform CRUD operations in your application (e.g., via admin panel or fixtures).
   • Run the command to generate a migration file:

     php bin/console appventus:datamigration:generate
     

   • Commit the generated migration file (e.g., migrations/20240101000000_EntityMigration.php) to version control.

Where to Look First

• Usage Documentation: Covers basic workflows (recording, generating, and applying migrations).
• Technical Documentation: Details event listeners, configuration, and customization.
• Generated Migration Files: Located in migrations/ (default path). Inspect these to understand recorded operations.

Implementation Patterns

Workflow: CMS Data Sync

1. Record Changes:

   • Developers or editors interact with the CMS (e.g., create/update/delete Article entities).
   • The bundle automatically logs these actions to a temporary storage (e.g., in-memory or session-based during the session).

2. Generate Migration:

   • After completing changes, run:

     php bin/console appventus:datamigration:generate
     

   • This creates a migration file (e.g., migrations/20240101000000_ArticleMigration.php) with serialized operations.

3. Share and Apply:

   • Commit the migration file to version control.
   • Team members pull the file and apply it to their local databases:

     php bin/console appventus:datamigration:migrate
     

Workflow: Fixture/Data Seeding

1. Seed Data:

   • Use your application’s logic to create test data (e.g., via a DataSeeder class).
   • Example:

     $user = new User();
     $user->name = 'Test User';
     $entityManager->persist($user);
     $entityManager->flush();
     

2. Capture Changes:

   • The bundle records the User creation in its internal log.

3. Generate and Reuse:

   • Generate the migration file:

     php bin/console appventus:datamigration:generate
     

   • Share the file with other developers or CI/CD pipelines to replicate the dataset.

Integration Tips

1. Custom Entities:

   • Exclude entities from tracking by annotating them with @ORM\ExcludeFromDataMigration or configuring the listener:

     # config/packages/appventus_data_migration.yaml
     appventus_data_migration:
         excluded_entities:
             - App\Entity\ExcludedEntity
     

2. Post-Processing:

   • Extend the migration file generation by implementing MigrationFileGeneratorInterface:

     class CustomMigrationGenerator implements MigrationFileGeneratorInterface {
         public function generate(MigrationData $data): string {
             // Custom logic (e.g., add timestamps or metadata)
             return $this->renderTemplate($data);
         }
     }
     

   • Register the service in services.yaml:

     AppVentus\DataMigrationBundle\MigrationFileGeneratorInterface: '@custom_migration_generator'
     

3. Batch Processing:

   • For large datasets, use the --batch-size option to split migrations:

     php bin/console appventus:datamigration:generate --batch-size=50
     

Gotchas and Tips

Pitfalls

1. Entity State Conflicts:

   • If an entity is updated multiple times in the same session, only the last operation is recorded. Use explicit flush() calls to separate logical operations.

2. Circular References:

   • The bundle serializes entities using Doctrine’s ObjectManager. Complex object graphs (e.g., bidirectional associations) may cause serialization issues. Test with:

     php bin/console appventus:datamigration:validate
     

3. Database Schema Mismatches:

   • Migrations assume the target database schema matches the source. Schema changes (e.g., new columns) will cause errors. Validate with:

     php bin/console doctrine:schema:validate
     

4. Performance Overhead:

   • Tracking every entity operation adds overhead. For high-traffic apps, disable tracking during bulk operations:

     $this->get('appventus_data_migration.listener')->disableTracking();
     // Perform bulk operations...
     $this->get('appventus_data_migration.listener')->enableTracking();
     

Debugging

1. Log Level:

   • Enable debug logs in config/packages/dev/appventus_data_migration.yaml:

     appventus_data_migration:
         debug: true
     

   • Check logs for recorded operations:

     bin/console debug:container appventus_data_migration.logger
     

2. Dry Runs:

   • Test migrations without applying them:

     php bin/console appventus:datamigration:migrate --dry-run
     

3. Migration File Inspection:

   • Generated files are PHP classes. Inspect them to verify operations:

     // Example generated migration:
     $this->create('App\Entity\User', [
         'id' => 1,
         'name' => 'Test User',
         'email' => 'test@example.com',
     ]);
     

Configuration Quirks

1. Default Migration Path:

   • Override the default migrations/ directory in config/packages/appventus_data_migration.yaml:

     appventus_data_migration:
         migration_directory: '%kernel.project_dir%/custom/migrations'
     

2. File Naming:

   • Migration files use the format YYYYMMDDHHmmss_EntityNameMigration.php. Customize the timestamp format via:

     appventus_data_migration:
         timestamp_format: 'YmdHis'
     

3. Dependency on ShortcutsBundle:

   • The package requires appventus/shortcuts-bundle. Install it explicitly if missing:

     composer require appventus/shortcuts-bundle:dev-master
     

Extension Points

1. Custom Migration Classes:

   • Override the default migration class by configuring a service alias:

     services:
         AppVentus\DataMigrationBundle\Migration\MigrationInterface: '@custom_migration_class'
     

2. Pre/Post-Migration Hooks:

   • Implement MigrationEventSubscriber to add logic before/after migration:

     class CustomMigrationSubscriber implements MigrationEventSubscriber {
         public function onPreMigrate(MigrationEvent $event) {
             // Add pre-migration logic (e.g., backups)
         }
     }
     

   • Register the subscriber in services.yaml:

     AppVentus\DataMigrationBundle\Event\MigrationEventSubscriberInterface: ['@custom_migration_subscriber', 'onPreMigrate']
     

3. Data Transformation:

   • Use MigrationDataTransformerInterface to modify recorded data before migration:

     class EmailObfuscatorTransformer implements MigrationDataTransformerInterface {
         public function transform(MigrationData $data) {
             foreach ($data->getEntities() as $entity) {
                 if ($entity instanceof User) {
                     $entity->email = 'obfuscated@example.com';
                 }
             }
             return $data;
         }
     }