Ezdataflow Bundle Laravel Package

## Getting Started

### Minimal Steps to Begin
1. **Installation**:
   ```bash
   composer require code-rhapsodie/ibexa-dataflow-bundle

Ensure CodeRhapsodie\DataflowBundle\CodeRhapsodieDataflowBundle and CodeRhapsodie\EzDataflowBundle\CodeRhapsodieEzDataflowBundle are enabled in config/bundles.php (Dataflow first).

2. Routing & DB Schema: Import routing in config/routing/ezdataflow.yaml:

   _cr.dataflow:
     resource: '@CodeRhapsodieEzDataflowBundle/Resources/config/routing.yaml'
   

   Run migrations as per DataflowBundle docs.

3. First Use Case: Define a DataflowType (e.g., ArticleImportDataflowType) tagged with coderhapsodie.dataflow.type:

   use CodeRhapsodie\DataflowBundle\DataflowType\AbstractDataflowType;
   use CodeRhapsodie\EzDataflowBundle\Writer\ContentWriter;
   
   class ArticleImportDataflowType extends AbstractDataflowType {
       public function __construct(ContentWriter $contentWriter) {
           $this->contentWriter = $contentWriter;
       }
       protected function buildDataflow(DataflowBuilder $builder, array $options) {
           $builder->addWriter($this->contentWriter);
       }
   }
   

   Tag it in services.yaml:

   App\Dataflow\ArticleImportDataflowType:
     tags: ['coderhapsodie.dataflow.type']
   

4. Access UI: Navigate to Admin > Ibexa Dataflow in the Ibexa backoffice to create schedules.

Implementation Patterns

Core Workflows

1. Content Import/Update: Use ContentWriter + ContentStructureFactory to transform data into ContentCreateStructure/ContentUpdateStructure:

   $builder->addStep(function ($data) {
       return $this->contentStructureFactory->transform(
           $data,
           'remote-id-' . $data['id'],
           'eng-GB',
           'article',
           123, // Parent location ID
           ContentStructureFactoryInterface::MODE_INSERT_OR_UPDATE
       );
   });
   

2. Scheduled Jobs:

   • Recurring: Define frequency (e.g., +1 day) and first run time in the UI.
   • One-shot: Set a fixed execution date via the "Oneshot" tab.

3. Data Validation: Use NotModifiedContentFilter to skip unchanged updates:

   $builder->addStep($this->notModifiedContentFilter);
   

Integration Tips

• Parent Locations: Store parent location IDs in .env.local (e.g., PARENT_ARTICLE_FOLDER=123).
• Field Mappings: Use ContentStructureFactory to map external fields to Ibexa field types (e.g., ezstring, ezrichtext).
• Error Handling: Log writer/reader errors via setLogger():

  $this->contentWriter->setLogger($this->logger);
  

Advanced Patterns

1. Custom Field Comparators: Extend AbstractFieldComparator for unsupported field types (e.g., ezmatrix):

   class MatrixFieldComparator extends AbstractFieldComparator {
       protected function compareValues(Value $current, Value $new): bool {
           return $current->value === $new->value;
       }
   }
   

   Register it in services.yaml:

   App\Comparator\MatrixFieldComparator:
     tags: ['coderhapsodie.ezdataflow.field_comparator', { fieldType: 'ezmatrix' }]
   

2. Dynamic Options: Pass runtime options via the UI (YAML format):

   api_url: https://example.com/data
   batch_size: 100
   

3. Queue Integration: Configure the dataflow_runner queue worker (e.g., Symfony Messenger or Supervisor):

   # config/packages/messenger.yaml
   framework:
       messenger:
           transports:
               dataflow: '%env(MESSENGER_TRANSPORT_DSN)%'
           routing:
               'CodeRhapsodie\DataflowBundle\Message\RunDataflowMessage': dataflow
   

Gotchas and Tips

Pitfalls

1. Bundle Loading Order:

   • Error: ClassNotFoundException for ContentWriter.
   • Fix: Ensure CodeRhapsodieDataflowBundle is loaded before CodeRhapsodieEzDataflowBundle in bundles.php.

2. Remote ID Conflicts:

   • Error: Duplicate remote_id values cause updates to fail.
   • Fix: Use unique prefixes (e.g., article-{id}) and validate in your DataflowType:

     if ($this->contentService->loadContentByRemoteId($remoteId)) {
         throw new \RuntimeException("Duplicate remote ID: $remoteId");
     }
     

3. Field Type Mismatches:

   • Error: InvalidArgumentException when mapping unsupported field types.
   • Fix: Implement custom comparators or exclude unsupported fields in transform().

4. Queue Stuck Jobs:

   • Error: Jobs remain in pending state.
   • Fix: Check queue worker logs and ensure the dataflow_runner is running:

     php bin/console messenger:consume dataflow -vv
     

5. Permission Issues:

   • Error: "Access Denied" in the UI.
   • Fix: Grant Setup / Administrate and Ibexa Dataflow / View policies to the user role.

Debugging Tips

1. Enable Logging: Configure Monolog in config/packages/monolog.yaml:

   handlers:
       dataflow:
           type: stream
           path: '%kernel.logs_dir%/dataflow.log'
           level: debug
   

2. Dump Dataflow Schema: Use the CLI command to inspect your DataflowType:

   php bin/console code-rhapsodie:dataflow:dump-schema --siteaccess=admin
   

3. Test Locally:

   • Use --dry-run to simulate jobs:

     php bin/console code-rhapsodie:dataflow:run --siteaccess=admin --dry-run
     

   • Validate ContentCreateStructure objects with:

     $this->contentStructureFactory->validate($structure);
     

Extension Points

1. Custom Writers: Extend AbstractWriter to support non-content operations (e.g., sending emails):

   class EmailWriter extends AbstractWriter {
       public function write(array $data) {
           // Send email logic
       }
   }
   

   Register it in your DataflowType:

   $builder->addWriter(new EmailWriter());
   

2. Pre/Post-Processing: Add steps for data enrichment or validation:

   $builder->addStep(function ($data) {
       $data['processed_at'] = new \DateTime();
       return $data;
   });
   

3. UI Customization: Override Twig templates in templates/bundles/CodeRhapsodieEzDataflowBundle/ to modify the admin interface.

4. Performance:

   • Batch Processing: Use Reader\BatchReader to limit memory usage:

     $builder->addReader(new BatchReader($reader, 50));
     

   • Indexing: Add ezsearch fields to ContentUpdateStructure for faster lookups:

     $structure->setField('search_keyword', $data['keyword']);
     

Configuration Quirks

1. Admin User:

   • Defaults to admin user. Override in config/packages/code_rhapsodie_ez_dataflow.yaml:

     code_rhapsodie_ez_dataflow:
         admin_login_or_id: webmaster
     

2. Siteaccess Awareness:

   • Use --siteaccess (not --connection) for CLI commands:

     php bin/console code-rhapsodie:dataflow:run --siteaccess=admin
     

3. YAML Options:

   • Complex options (e.g., nested arrays) must use full YAML syntax in the UI:

     options:
         filters:
             - { field: status, value: published }
             - { field: date, operator: '>', value: '2023-01-01' }
     

Pro Tips

1. Environment-Specific Configs: Use parameters.yaml for environment-specific settings (e.g., API keys):

   parameters:
       dataflow:
           api_key: '%env(APP_API_KEY)%'
   

2. Reusable DataflowTypes: Share DataflowType classes across projects via Composer packages.