Parallel Laravel Package

Technical Evaluation

Architecture Fit

• Event-Driven & Async-First Fit: amphp/parallel aligns with Laravel’s growing adoption of async/await (via Symfony’s Fiber and Amp) and event-driven architectures (e.g., queues, jobs). It bridges blocking I/O (e.g., file_get_contents, CPU-heavy tasks) with non-blocking event loops, reducing latency in high-concurrency scenarios.
• Worker Pool Abstraction: The WorkerPool interface abstracts process/thread management, making it ideal for Laravel’s job queues (e.g., Illuminate\Bus\Queue) or batch processing (e.g., Illuminate\Support\Facades\Bus::batch). Tasks can be distributed across workers without manual process orchestration.
• Serialization Constraints: Tasks must be serializable (via Task interface), which may conflict with Laravel’s dependency-injected services (e.g., repositories, Eloquent models). Workarounds include:
  • DTOs: Convert services to data transfer objects (DTOs) for task payloads.
  • Static Caching: Use LocalCache/AtomicCache for shared state (as shown in the docs), but avoid Laravel’s service container (which isn’t serializable).
• Hybrid Sync/Async: Laravel’s sync queues (e.g., sync:work) can leverage amphp/parallel for parallel task execution within a single request, while async queues (e.g., database, redis) remain unchanged.

Integration Feasibility

• Laravel Compatibility:
  • PHP 8.1+: No conflicts with Laravel’s minimum version (8.2+).
  • Amp Integration: Laravel’s illuminate/support includes Amp\Loop (since v9.x), enabling seamless event loop integration. However, Laravel’s default Swoole/ReactPHP servers may require adjustments to delegate to Amp’s loop.
  • Queue Workers: Replace Laravel’s queue workers (php artisan queue:work) with a custom worker using WorkerPool to process jobs in parallel. Example:

    $pool = new WorkerPool(new ProcessContextFactory(), 4); // 4 workers
    $pool->submit(new DatabaseJobTask($job));
    

  • Middleware/Requests: Use Worker::submit() in middleware or controllers for blocking operations (e.g., PDF generation, API calls to external services) without blocking the HTTP request.
• Database Considerations:
  • Connection Handling: Database connections (e.g., PDO, Eloquent) are not serializable. Each worker must establish its own connection. Use connection pooling (e.g., pdo_pgsql/pdo_mysql with persistent mode) or lazy-load connections in Task::run().
  • Transactions: Distributed transactions across workers are unsupported. Use saga patterns or compensating transactions for cross-worker data consistency.

Technical Risk

• Complexity Overhead:
  • Learning Curve: Amp’s fiber/parallel paradigm differs from Laravel’s traditional request lifecycle. Developers must understand Task interfaces, Channel IPC, and cancellation patterns.
  • Debugging: Worker crashes or deadlocks (e.g., unhandled exceptions in Task::run()) may require custom error handling or logging (e.g., WorkerPool::onError()).
• Performance Pitfalls:
  • Serialization Overhead: Large task payloads (e.g., complex objects) may serialize slowly. Optimize with DTOs or primitive types.
  • Process vs. Threads: Threads (via ext-parallel) offer lower latency but require PHP 8.2+ ZTS. Processes are more portable but slower to spawn.
• Laravel-Specific Risks:
  • Service Container: Laravel’s container is not serializable. Avoid injecting it into tasks; use constructor injection with DTOs or static methods.
  • Event System: Laravel’s events (e.g., Event::dispatch()) won’t propagate to workers. Use Channel IPC or a shared message bus (e.g., Redis) for cross-worker communication.
  • Testing: Unit testing tasks requires mocking Channel and Cancellation. Integration tests may need custom WorkerPool setups.

Key Questions

1. Use Case Alignment:
   • Are we targeting CPU-bound tasks (e.g., image processing) or I/O-bound tasks (e.g., external API calls)? Threads suit CPU; processes suit I/O.
   • Will tasks require shared state (e.g., caching)? If so, how will we handle AtomicCache vs. LocalCache?
2. Deployment Constraints:
   • Can we use threads (ext-parallel) or must we rely on processes? Threads reduce latency but require PHP 8.2+ ZTS.
   • How will we handle worker crashes? Will we use Laravel’s queue retries or custom WorkerPool error handlers?
3. Integration Points:
   • Should we replace Laravel queues entirely or use amphp/parallel for specific jobs (e.g., high-priority batch jobs)?
   • How will we integrate with Laravel’s HTTP layer? Will we use middleware or a custom Amp server (e.g., Amp\Http\Server)?
4. Monitoring:
   • How will we monitor worker health? Metrics (e.g., task duration, failures) should be exposed via WorkerPool extensions.
   • Can we leverage Laravel Horizon for worker management, or will we need a custom dashboard?

Integration Approach

Stack Fit

• Laravel Core:
  • Event Loop: Replace Laravel’s default ReactPHP/Swoole loop with Amp\Loop in custom server setups (e.g., CLI commands, queues). Use Amp\Loop::run() in artisan commands or queue workers.
  • Queues: Extend Illuminate\Contracts\Queue\Job to support Task interface. Example:

    class ParallelJob implements Task, ShouldQueue {
        public function run(Channel $channel, Cancellation $cancellation) { ... }
        public function handle() { ... } // Legacy Laravel queue method
    }
    

  • HTTP: Use Amp\Http\Server for async routes (e.g., WebSockets, long-polling) alongside Laravel’s router. For sync routes, offload blocking tasks to workers via middleware.
• Dependencies:
  • Amp: Required for event loop and fibers. Conflict risk with reactphp/react (Laravel’s default). Resolve via composer overrides or custom bootstrap.
  • Ext-Parallel: Only needed for threads. Ensure PHP 8.2+ ZTS build if targeting thread-based workers.
  • Cluster: Optional for advanced IPC (e.g., cross-process sockets). Useful for microservices or distributed setups.

Migration Path

1. Phase 1: Proof of Concept (PoC)
   • Scope: Replace a single blocking operation (e.g., file_get_contents in a job) with Worker::submit().
   • Implementation:
     • Create a Task class for the blocking operation.
     • Test in a CLI command or queue job.
     • Measure performance vs. sync/blocking baseline.
   • Tools: Use Amp\Loop::run() in a custom artisan command for testing.
2. Phase 2: Worker Pool Integration
   • Scope: Replace Laravel’s queue workers with a WorkerPool.
   • Implementation:
     • Extend Illuminate\Queue\Worker to use WorkerPool for job execution.
     • Configure worker count based on server resources (e.g., 4 * CPU cores).
     • Add error handling for failed tasks (e.g., log to Laravel’s logs table).
   • Example:

     $pool = new WorkerPool(new ProcessContextFactory(), config('queue.workers'));
     $pool->submit(new DatabaseJobTask($job));
     

3. Phase 3: HTTP Layer Integration
   • Scope: Offload blocking middleware/route logic to workers.
   • Implementation:
     • Create middleware that submits tasks to a global WorkerPool.
     • Use Future\await() to block only the worker, not the HTTP request.
     • Example:

       public function handle($request, Closure $next) {
           $task = new ImageProcessingTask($request->file('image'));
           $future = Worker\submit($task);
           $result = $future->await(); // Blocks only the worker
           return $next($request->merge(['processed_image' => $result]));
       }
       

4. Phase 4: Full Async Stack
   • Scope: Replace ReactPHP/Swoole with Amp for HTTP and queues.
   • Implementation:
     • Use Amp\Http\Server for async routes.
     • Migrate queue workers to WorkerPool.
     • Replace reactphp/event-loop with amphp/amp in config/app.php.
   • Risk: High complexity; consider gradual migration.

Compatibility

• Laravel Services:
  • Non-Serializable Services: Avoid injecting Illuminate\Contracts\Container,