Weave Code
Code Weaver
Helps Laravel developers discover, compare, and choose open-source packages. See popularity, security, maintainers, and scores at a glance to make better decisions.
Feedback
Share your thoughts, report bugs, or suggest improvements.
Subject
Message

Pipeline Laravel Package

amphp/pipeline

Fiber-safe concurrent iterators and collection operators for AMPHP. Build pipelines from iterables, map/filter/merge, and consume results from multiple fibers safely using ConcurrentIterator (foreach or manual continue/getValue/getPosition). Requires PHP 8.1+.

View on GitHub
Deep Wiki
Context7

Technical Evaluation

Architecture Fit

  • Event-Driven & Async-First: amphp/pipeline is a natural fit for Laravel applications leveraging AMPHP (e.g., amphp/amp, amphp/http-client) or those requiring fiber-based concurrency (e.g., high-throughput APIs, real-time processing). It aligns with Laravel’s growing support for Swoole and RoadRunner (which use fibers).
  • Functional Pipeline Pattern: Enables declarative, composable data processing (e.g., map/filter/reduce) akin to Laravel’s collection methods, but with asynchronous guarantees. Ideal for:
    • Batch processing (e.g., queued jobs, bulk API calls).
    • Streaming data (e.g., WebSocket messages, database cursors).
    • Concurrent I/O-bound tasks (e.g., parallel HTTP requests, database queries).
  • Back-Pressure Support: Prevents memory bloat by throttling producers when consumers lag (critical for Laravel’s queue workers or event listeners).

Integration Feasibility

  • Laravel Compatibility:
    • AMPHP Integration: Works seamlessly with Laravel’s AMPHP integration (e.g., spatie/laravel-amp) or Swoole (via laravel/swoole).
    • Queue Workers: Replace Illuminate\Queue\Worker loops with fiber-based Queue consumers for non-blocking processing.
    • Event Loop: Integrates with Laravel’s event loop (e.g., symfony/event-dispatcher + AMPHP) for async listeners.
  • Existing Laravel Patterns:
    • Collections: Replace Collection::pipe() with Pipeline::fromIterable() for async operations.
    • Jobs: Use Pipeline::concurrent() to parallelize ShouldQueue jobs.
    • Database: Stream results from DB::cursor() or Eloquent batches via Pipeline.
  • PHP 8.1+ Requirement: Aligns with Laravel’s PHP 8.1+ roadmap, but may require runtime upgrades (e.g., Swoole 4.6+ or RoadRunner).

Technical Risk

Risk Area Mitigation Strategy
Fiber Adoption Laravel’s core does not natively support fibers; requires Swoole/RoadRunner.
Blocking Legacy Code Wrap synchronous code in Amp\async() or use Pipeline::buffer() to avoid deadlocks.
State Management ConcurrentIterator must be properly disposed (e.g., in finally blocks) to avoid memory leaks.
Debugging Complexity Use tap() for logging/debugging in pipelines; AMPHP’s Amp\Loop::run() helps isolate issues.
Vendor Lock-in AMPHP ecosystem is small; prefer interfaces (ConcurrentIterator) over concrete classes.

Key Questions

  1. Concurrency Model:
    • Will the app use Swoole, RoadRunner, or AMPHP standalone? This dictates fiber support.
    • How will Pipeline interact with Laravel’s synchronous services (e.g., Eloquent, Cache)?
  2. Error Handling:
    • How will pipeline failures (e.g., DisposedException) map to Laravel’s exception handlers?
    • Should Pipeline::error() integrate with Laravel’s App\Exceptions\Handler?
  3. Performance Trade-offs:
    • What’s the optimal concurrent() batch size for I/O-bound vs. CPU-bound tasks?
    • How will buffer() affect memory usage in long-running pipelines?
  4. Testing:
    • How to mock ConcurrentIterator in PHPUnit (e.g., for unit tests)?
    • Tools for stress-testing back-pressure (e.g., Queue + delay()).
  5. Deployment:
    • Does the hosting environment support PHP fibers (e.g., shared hosting may not)?
    • How to handle graceful shutdowns (e.g., Queue::complete() on SIGTERM)?

Integration Approach

Stack Fit

Laravel Component Integration Strategy
Queue Workers Replace Illuminate\Queue\Worker with a fiber-based Queue consumer. Example:
```php
Amp\Loop::run(function () {
$queue = new Queue();
async(function () use ($queue) {
while ($job = dispatch()->get()) {
$queue->push($job->handle());
}
$queue->complete();
});
foreach ($queue->iterate() as $result) {
Log::info("Processed: {$result}");
}
});
```
HTTP Requests Parallelize Http::async() calls using Pipeline::concurrent().
Database Stream Eloquent batches or DB::cursor() results through Pipeline::fromIterable().
Events Replace Event::dispatch() with Pipeline::merge() for async event aggregation.
Commands Use Pipeline in Artisan commands for async CLI tasks (e.g., imports).
Service Providers Bind Pipeline to Laravel’s container for dependency injection.

Migration Path

  1. Phase 1: Proof of Concept
    • Replace a synchronous batch job (e.g., Artisan::call('queue:work')) with a Pipeline-based fiber consumer.
    • Example: Convert a foreach loop over a collection to Pipeline::fromIterable().
  2. Phase 2: Core Integration
    • Wrap Laravel’s queue system in a fiber-aware layer (e.g., custom Queue class).
    • Add Pipeline methods to Laravel’s Collection facade (via traits or macros).
  3. Phase 3: Full Adoption
    • Migrate I/O-bound services (e.g., HTTP clients, database) to use Pipeline.
    • Replace Process facade with Amp\Process for async subprocesses.
  4. Fallback Strategy
    • Use Amp\async() for synchronous code paths to avoid blocking fibers.
    • Implement circuit breakers for pipeline failures (e.g., retry logic).

Compatibility

  • Laravel 9+: Native support for PHP 8.1+; leverage spatie/laravel-amp for AMPHP.
  • Swoole/RoadRunner: Required for fiber execution; configure Laravel’s APP_RUN_IN_CONSOLE or SERVER_SOFTWARE checks.
  • Existing Packages:
    • Queues: Works with database, redis, or beanstalkd queues via Queue consumers.
    • HTTP: Integrates with guzzlehttp/guzzle via Amp\Http\Client.
    • Events: Use Pipeline::merge() to combine multiple event sources.
  • Database:
    • Eloquent: Stream results with Model::cursor()Pipeline::fromIterable().
    • Query Builder: Use DB::statement() with generators for async SQL.

Sequencing

  1. Critical Path:
    • Start with non-critical async tasks (e.g., logging, analytics) to validate fiber safety.
    • Gradually move to core workflows (e.g., order processing, real-time updates).
  2. Dependency Order:
    • Step 1: Set up AMPHP/Swoole environment.
    • Step 2: Replace blocking I/O (e.g., file_get_contents()Amp\File\read()).
    • Step 3: Introduce Pipeline for data processing.
    • Step 4: Migrate queue workers to fiber-based consumers.
  3. Rollback Plan:
    • Use feature flags to toggle Pipeline usage.
    • Maintain synchronous fallbacks for unsupported environments.

Operational Impact

Maintenance

  • Pros:
    • Reduced Boilerplate: Pipeline operators replace manual foreach/yield loops.
    • Consistent Error Handling: Centralize exception handling in Pipeline::error().
    • Back-Pressure Safety: Prevents OOM crashes in long-running processes.
  • Cons:
    • Debugging Complexity: Fibers and async flows require new tooling (e.g., Xdebug 3+).
    • State Management: ConcurrentIterator disposal must be explicit (risk of leaks).
    • Vendor Dependencies: AMPHP ecosystem is smaller than Laravel’s; require deep expertise.

Support

  • Monitoring:
    • Track Queue back-pressure metrics (e.g., `push
Weaver

How can I help you explore Laravel packages today?

Conversation history is not saved when not logged in.
Prompt
Add packages to context
No packages found.
calmfox/watch-sylius
damienfern/grpc-symfony-bundle
atoolo/index-bundle
atoolo/genai-bundle
coprotoai/laravel-ticket
davidjln/llm-carbon-bundle
cryonighter/valid-request-bundle
coolms/taxonomy-bundle
coolms/field-bundle
articulate-orm/symfony
aaix/laravel-tall-architect
ephoto/akeneo-connector
emmanuelballery/eb-plantumlbundle
emielburgman/symfony-visitor-beacon
emielburgman/symfony-visit-storage
emielburgman/symfony-security-headers
emielburgman/symfony-log-viewer
emarref/xdebug-bundle
emarref/pubnub-bundle
elriseio/finance-money-bundle