Weave Code
Data Processor Laravel Package
derafu/data-processor
Laravel/PHP data processing toolkit for defining, transforming, validating, and exporting datasets through a consistent pipeline. Provides reusable processors and helpers to normalize inputs, run rules, and produce clean output for apps, imports, and integrations.
Four-Phase Data Processing Library
Frequently asked questions about Data Processor
- How does the four-phase pipeline (ingestion → transformation → validation → output) align with Laravel’s architecture?
- The package is designed to fit Laravel’s service layer and event-driven patterns. Each phase can be treated as a modular service, with ingestion triggering Laravel queues for async processing, transformations leveraging Collections, and validation integrating with Laravel’s built-in Validator. Phase transitions can emit Laravel events for observability.
- Can I use this package for async CSV imports in Laravel without blocking the request?
- Yes, the package supports async processing by wrapping phases in Laravel Jobs. You can dispatch ingestion or transformation phases as queue jobs, leveraging Laravel’s queue system (Redis, database, etc.) for scalability. The output phase can then write results to storage or trigger notifications.
- Does Derafu Data Processor support Laravel’s built-in validation rules (e.g., `Validator::make()`) or does it require custom logic?
- The package integrates with Laravel’s Validator by default, allowing you to reuse existing validation rules or extend it with custom logic. You can pass Laravel’s validation rules directly in the validation phase, or use the package’s native validator for simpler cases. It also supports FormRequest validation for API endpoints.
- What Laravel versions and PHP requirements does this package support?
- The package is compatible with Laravel 10.x and 11.x, requiring PHP 8.1+. Always check the package’s `composer.json` for the latest constraints, as minor updates may introduce version-specific optimizations or dependency changes.
- How do I handle database transactions when processing data in phases? Does the package manage them automatically?
- The package does not enforce transactions by default, allowing you to wrap phases in Laravel’s database transactions manually. For example, you can use `DB::transaction()` around the ingestion or transformation phases if atomicity is required. Async jobs should handle their own transactions via Laravel’s `ShouldQueue` interface.
- Can I extend or override individual phases (e.g., transformation) without modifying the core package?
- Yes, the package follows a modular design where phases can be overridden or extended via dependency injection. You can replace the default transformation logic with your own class by binding it to the service container, ensuring backward compatibility with the rest of the pipeline.
- Are there any known conflicts with popular Laravel packages like `spatie/array-to-object` or `league/csv`?
- The package is designed to avoid tight coupling with Laravel-specific classes, reducing dependency conflicts. However, if you use `league/csv` for ingestion or `spatie/array-to-object` for transformation, ensure their versions are compatible with the package’s PHP 8.1+ requirements. Test in a staging environment if mixing with other data-processing packages.
- How do I test this package in a Laravel application, especially for queue jobs or database interactions?
- Test each phase independently using Laravel’s testing tools. For queue jobs, use `Queue::fake()` to assert job dispatching, and for database interactions, leverage `DatabaseMigrations` or `DatabaseTransactions`. The package’s documentation should include Laravel-specific test examples, such as mocking data sources or validating output formats.
- What’s the best approach to migrate from custom ETL logic to this package in an existing Laravel app?
- Start with a proof of concept by replacing one phase (e.g., transformation) in a non-critical workflow. Use mock data (e.g., a CSV file) to validate the output matches your expectations. Gradually migrate other phases, aligning validation rules with Laravel’s Validator and wrapping async phases in Jobs for background processing.
- Is this package suitable for real-time data processing, or is it better for batch operations?
- The package is optimized for batch operations and structured pipelines, not real-time processing. For low-latency systems, consider Laravel’s built-in Jobs or middleware for ad-hoc transformations. Async processing (via queues) is supported but requires explicit configuration—ideal for imports, report generation, or API data enrichment.
Weaver
How can I help you explore Laravel packages today?