Weave Code
ETL Core lib
Frequently asked questions about Etl Core Bundle
- Can I use ETLCoreBundle in Laravel, or is it strictly for Symfony?
- ETLCoreBundle is designed for Symfony but can be adapted for Laravel with effort. You’ll need to bridge Symfony’s Messenger (for CQS) with Laravel’s queues (e.g., Redis or database) and handle Doctrine ORM dependencies—either by using a hybrid Eloquent/Doctrine setup or wrapping Symfony services in Laravel-compatible facades. The bundle’s core pipeline logic (steps, workflows) remains framework-agnostic.
- What’s the easiest way to integrate this with Laravel’s Eloquent instead of Doctrine?
- Since the bundle requires Doctrine entities for workflow/pipeline state, you’ll need to either: 1) Use Doctrine *only* for ETL entities while keeping Eloquent for app models, or 2) Create a custom repository layer that translates Doctrine queries to Eloquent. The bundle’s abstract `Pipeline` and `Step` classes can be extended to work with Eloquent, but state machine logic (e.g., `pending` → `completed`) relies on Doctrine events.
- How do I handle asynchronous execution in Laravel (e.g., queues) instead of Symfony Messenger?
- Replace Symfony Messenger with Laravel’s queues by creating a facade or adapter that dispatches pipeline steps to Laravel’s queue system. The bundle’s `ExecutableStep` interface can be implemented to defer execution to a queue job. For workflow state transitions (e.g., retries), you’ll need to manually trigger Laravel events or use a custom state machine library like `spatie/laravel-workflow` to mirror Symfony’s Workflow component.
- Does this bundle support scheduled ETL pipelines (e.g., cron jobs)?
- Yes, but with Symfony’s scheduling tools. For Laravel, you’d need to integrate Symfony’s `CronTrigger` or `EventTrigger` with Laravel’s task scheduler (e.g., `laravel-scheduler`). The bundle’s `Pipeline` entity tracks execution history, so you can query completed/failed runs. For cron-like scheduling, use Laravel’s `schedule:run` command or a Symfony cron job that triggers a Laravel HTTP endpoint.
- Are there pre-built steps for common ETL tasks like CSV parsing or API calls?
- The bundle provides abstract `ExecutableStep` classes and integrates with **Flow-PHP ETL**, which includes built-in steps for CSV, JSON, and database operations. However, you’ll need to register and configure these steps as tagged services in Symfony’s DI container. For Laravel, you’d either replicate these steps or use Laravel packages like `league/csv` or `guzzlehttp/guzzle` alongside the bundle’s pipeline engine.
- How do I test pipelines with custom steps in Laravel’s testing environment?
- Mock Symfony services (e.g., `Workflow`, `Messenger`) using Laravel’s `Mockery` or PHPUnit’s `createMock()`. For Doctrine dependencies, use Laravel’s `DatabaseMigrations` or `RefreshDatabase` traits to seed test data. Test steps in isolation by injecting a minimal `PipelineContext` and verifying output. For workflow transitions, mock the state machine or use a lightweight alternative like `spatie/laravel-workflow` in tests.
- What’s the performance impact of using Symfony’s Workflow for pipeline state management?
- Symfony’s Workflow adds minimal overhead for state transitions (e.g., `pending` → `in_progress`), but it’s not optimized for high-throughput ETL. For production, batch steps to avoid memory issues and offload long-running tasks to Laravel queues. If performance is critical, consider replacing the state machine with a simpler solution (e.g., a database `status` column) or using a dedicated ETL tool like **Spatie’s Laravel ETL** for simpler pipelines.
- Can I customize workflow transitions (e.g., add retries or timeouts) beyond the defaults?
- Yes, extend the bundle’s `Workflow` class to override transition logic. For retries, configure the `retryPolicy` in the workflow definition or implement a custom `ExecutableStep` that handles failures. Timeouts can be set per step via the `timeout` configuration. For advanced use cases, integrate with Symfony’s `Workflow` component directly or use a Laravel event listener to intercept pipeline state changes.
- How do I monitor pipeline execution in Laravel (e.g., logs, metrics, or a dashboard)?
- Leverage Laravel’s logging (Monolog) to track step execution and errors. For metrics, use Laravel Horizon or a package like `spatie/laravel-monitoring` to visualize pipeline status. The bundle’s `Pipeline` entity stores timestamps and status, so you can query these via Eloquent/Doctrine. For auditing, add a `log` table to record step inputs/outputs or integrate with Symfony’s Profiler (if using a hybrid setup).
- What are the alternatives to ETLCoreBundle for Laravel ETL, and when should I choose them?
- For Laravel, consider **Spatie’s Laravel ETL** (simpler, Eloquent-focused) if you need basic pipelines without Symfony dependencies. **Laravel Jobs + Queues** work for lightweight ETL but lack workflow state management. **Airflow** or **Prefect** are better for orchestration-heavy workflows. Choose ETLCoreBundle if you need DDD alignment, CQS, or Symfony’s Workflow for complex stateful pipelines—especially if your app already uses Symfony components or requires advanced scheduling.
Weaver
How can I help you explore Laravel packages today?