Ibexa Dataflow Bundle Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require code-rhapsodie/ibexa-dataflow-bundle
   

   Add bundles to config/bundles.php (ensure CodeRhapsodie\DataflowBundle loads first):

   CodeRhapsodie\DataflowBundle\CodeRhapsodieDataflowBundle::class => ['all' => true],
   CodeRhapsodie\IbexaDataflowBundle\CodeRhapsodieIbexaDataflowBundle::class => ['all' => true],
   

2. Routing: Import routes in config/routing/ibexa_dataflow.yaml:

   _cr.ibexa_dataflow:
     resource: '@CodeRhapsodieIbexaDataflowBundle/Resources/config/routing.yaml'
   

3. Database & Queue: Run migrations and configure the queue worker (see DataflowBundle docs).

4. First Dataflow: Define a DataflowType (see DataflowBundle docs). Tag it with coderhapsodie.dataflow.type in services.yaml:

   App\Dataflow\MyDataflowType:
     tags: ['coderhapsodie.dataflow.type']
   

5. Access UI: Navigate to Admin > Ibexa Dataflow in the Ibexa backoffice to create schedules or one-shot jobs.

Implementation Patterns

Core Workflow: Content Import/Export

1. Define a DataflowType: Extend AbstractDataflowType and implement buildDataflow() to chain readers, steps, and writers. Example:

   class ArticleImportDataflow extends AbstractDataflowType {
       protected function buildDataflow(DataflowBuilder $builder, array $options) {
           $builder
               ->addReader(new CsvReader($options['file_path']))
               ->addStep($this->contentStructureFactory->createStep($options))
               ->addWriter(new ContentWriter());
       }
   }
   

2. Content Transformation: Use ContentStructureFactory to convert raw data into ContentCreateStructure/ContentUpdateStructure:

   $factory->transform(
       $data,          // Raw data array
       'remote-id',    // Unique identifier (e.g., 'article-123')
       'eng-GB',       // Language code
       'article',      // Content type identifier
       42,             // Parent location ID
       ContentStructureFactoryInterface::MODE_INSERT_OR_UPDATE
   );
   

3. Scheduled Jobs:

   • Recurring: Configure via UI with strtotime-compatible intervals (e.g., +1 day).
   • One-shot: Set a specific execution date/time in the "Oneshot" tab.

4. Filtering: Skip unchanged content with NotModifiedContentFilter:

   $builder->addStep(new NotModifiedContentFilter());
   

5. Error Handling: Log exceptions via setLogger() on writers/filters:

   $writer->setLogger($this->logger);
   

Integration Tips

1. Dynamic Parent Locations: Fetch parent location IDs dynamically using LocationService:

   $parentLocation = $locationService->loadLocation($parentLocationId);
   

2. Custom Field Mappings: Override ContentStructureFactory to handle non-standard fields:

   class CustomContentStructureFactory extends ContentStructureFactory {
       protected function mapField($fieldName, $data) {
           // Custom logic for fields like 'custom_matrix'
       }
   }
   

3. Bulk Operations: Use ContentWriter::setBatchSize() to optimize performance for large imports:

   $writer->setBatchSize(50); // Process 50 items per batch
   

4. Validation: Add validation steps before content creation:

   $builder->addStep(function ($data) {
       if (empty($data['title'])) {
           throw new \RuntimeException('Title is required.');
       }
       return $data;
   });
   

5. Environment-Specific Config: Store sensitive values (e.g., API keys, parent location IDs) in .env.local:

   # config/packages/code_rhapsodie_ibexa_dataflow.yaml
   code_rhapsodie_ibexa_dataflow:
       admin_login_or_id: '%env(IBEXA_DATAFLOW_USER)%'
   

Gotchas and Tips

Pitfalls

1. Loading Order:

   • Error: ClassNotFoundException for ContentWriter.
   • Fix: Ensure CodeRhapsodie\DataflowBundle loads before CodeRhapsodie\IbexaDataflowBundle in bundles.php.

2. Remote ID Conflicts:

   • Issue: Duplicate remoteId values cause updates to overwrite unrelated content.
   • Solution: Use a unique prefix (e.g., source-article-123) and validate in your step:

     if ($contentService->loadContentByRemoteId($remoteId)) {
         throw new \RuntimeException("Duplicate remote ID: $remoteId");
     }
     

3. Field Type Mismatches:

   • Error: InvalidArgumentException when comparing unsupported field types (e.g., ibexa_image).
   • Fix: Exclude unsupported fields or implement a custom FieldComparator (see below).

4. Queue Stuck Jobs:

   • Debug: Check dataflow_job table for stuck jobs. Manually set status = 'failed' to retry.
   • Fix: Ensure the queue worker is running (php bin/console cr:dataflow:run).

5. Permission Issues:

   • Error: "Access Denied" when running dataflows.
   • Fix: Grant the configured admin_login_or_id user the Content and Location permissions.

Debugging Tips

1. Enable Logging: Configure Monolog in config/packages/monolog.yaml:

   handlers:
       dataflow:
           type: stream
           path: "%kernel.logs_dir%/dataflow.log"
           level: debug
   

2. Dry Runs: Use ContentStructureFactory::MODE_INSERT_ONLY to test without creating content:

   $factory->transform(..., ContentStructureFactoryInterface::MODE_INSERT_ONLY);
   

3. SQL Queries: Enable Doctrine profiling to inspect queries:

   $this->container->get('profiler')->disable();
   $this->container->get('profiler')->enable();
   

4. Dataflow Dump: Inspect raw data before processing:

   php bin/console cr:dataflow:dump-schema --siteaccess=admin
   

Extension Points

1. Custom Field Comparators: Extend AbstractFieldComparator for unsupported field types (e.g., ibexa_image):

   class ImageFieldComparator extends AbstractFieldComparator {
       protected function compareValues(Value $current, Value $new) {
           return $current->value === $new->value; // Simple comparison
       }
   }
   

   Register the service:

   App\FieldComparator\ImageFieldComparator:
       tags:
           - { name: 'coderhapsodie.ibexa_dataflow.field_comparator', fieldType: 'ibexa_image' }
   

2. Custom Writers: Implement DataflowWriterInterface for non-content operations (e.g., logging to an external API):

   class ApiWriter implements DataflowWriterInterface {
       public function write(array $data) {
           $client->post('/api/items', ['data' => $data]);
       }
   }
   

3. Pre/Post Hooks: Add steps for side effects (e.g., sending notifications):

   $builder->addStep(function ($data) after: 'content_writer') {
       $this->notificationService->send($data['email']);
   });
   

4. Dynamic Dataflow Types: Load DataflowType classes dynamically from a directory:

   # config/services.yaml
   App\Dataflow\:
       resource: '../src/Dataflow/*Dataflow.php'
       tags: ['coderhapsodie.dataflow.type']
   

5. UI Customization: Override Twig templates in templates/bundles/CodeRhapsodieIbexaDataflow/ to modify the admin UI.