Rolling Curl Laravel Package

Getting Started

Minimal Steps

1. Installation:

   composer require chuyskywalker/rolling-curl
   

   Add to composer.json under require:

   "chuyskywalker/rolling-curl": "*"
   

2. Basic Usage:

   use RollingCurl\RollingCurl;
   
   $rollingCurl = new RollingCurl();
   $rollingCurl->get('https://example.com/api/1')
               ->get('https://example.com/api/2')
               ->setSimultaneousLimit(5)
               ->setCallback(function($request, $rollingCurl) {
                   // Process response here
               })
               ->execute();
   

3. First Use Case: Replace a blocking file_get_contents() loop with RollingCurl for fetching multiple API endpoints or web pages concurrently. Example:

   $urls = ['https://api.example.com/users', 'https://api.example.com/posts'];
   $rollingCurl = new RollingCurl();
   foreach ($urls as $url) {
       $rollingCurl->get($url);
   }
   $rollingCurl->setSimultaneousLimit(3)->execute();
   

Where to Look First

• Examples Directory: Clone the repo and inspect examples/ for real-world patterns (e.g., scraping, API polling).
• Class Methods: Focus on:
  • get(), post(), put(), delete() for request types.
  • setSimultaneousLimit() to control concurrency.
  • setCallback() for per-request processing.
  • execute() to start the rolling queue.

Implementation Patterns

Usage Patterns

1. Request Chaining

Chain requests fluently before execution:

$rollingCurl = new RollingCurl();
$rollingCurl->get('https://api.example.com/data')
            ->post('https://api.example.com/submit', ['key' => 'value'])
            ->setSimultaneousLimit(5);

2. Per-Request Callbacks

Process responses as they complete (avoids memory buildup):

$rollingCurl->setCallback(function($request, $rollingCurl) {
    $data = json_decode($request->getResponseText(), true);
    // Store in DB, queue a job, etc.
    $rollingCurl->clearCompleted(); // Free memory
});

3. Bulk URL Fetching

Loop through URLs with dynamic options:

$urls = ['url1', 'url2', 'url3'];
foreach ($urls as $url) {
    $request = new \RollingCurl\Request($url);
    $request->addOptions([CURLOPT_TIMEOUT => 10]);
    $rollingCurl->add($request);
}

4. Laravel Integration

• Jobs: Wrap execute() in a Job to avoid blocking:

  use Illuminate\Bus\Queueable;
  use RollingCurl\RollingCurl;
  
  class FetchUrlsJob extends Job {
      use Queueable;
      public function handle() {
          $rollingCurl = new RollingCurl();
          // ... add requests ...
          $rollingCurl->execute();
      }
  }
  

• Service Provider: Bind RollingCurl to Laravel’s container:

  $this->app->singleton(RollingCurl::class, function() {
      return new RollingCurl();
  });
  

5. Error Handling

Catch failures in the callback:

$rollingCurl->setCallback(function($request) {
    if ($request->getError()) {
        Log::error("Failed: " . $request->getError());
        // Retry logic or dead-letter queue
    }
});

Workflows

Scraping Workflow

1. Queue URLs: Add all target URLs to RollingCurl.
2. Set Limits: Use setSimultaneousLimit(10) to avoid DOS.
3. Process Responses: Extract data in the callback (e.g., parse HTML with DOMDocument).
4. Store Results: Use Laravel’s Model::create() or queue a StoreScrapedDataJob.

API Polling Workflow

1. Dynamic Requests: Loop through API endpoints with pagination:

   for ($page = 1; $page <= 10; $page++) {
       $rollingCurl->get("https://api.example.com/data?page=$page");
   }
   

2. Aggregate Data: Use a shared variable in the callback to accumulate results.
3. Rate Limiting: Respect Retry-After headers by pausing the queue:

   $rollingCurl->setCallback(function($request) {
       if ($request->getResponseHeader('Retry-After')) {
           sleep((int) $request->getResponseHeader('Retry-After'));
       }
   });
   

Background Processing

• Artisan Command:

  Artisan::command('scrape:urls', function() {
      $rollingCurl = new RollingCurl();
      // ... add requests ...
      $rollingCurl->execute();
  });
  

• Schedule: Run via Laravel’s scheduler (schedule:run).

Integration Tips

1. Avoid Memory Leaks:

   • Call clearCompleted() and prunePendingRequestQueue() in the callback.
   • For large batches, process results in chunks (e.g., every 100 requests).

2. Laravel HTTP Client Bridge: If using Laravel’s HttpClient, convert RollingCurl responses to Illuminate\Http\Client\Response:

   $rollingCurl->setCallback(function($request) {
       $response = new \Illuminate\Http\Client\Response(
           $request->getResponseText(),
           $request->getStatusCode(),
           $request->getResponseHeaders()
       );
       // Use Laravel's response methods (e.g., $response->json())
   });
   

3. Testing:

   • Mock RollingCurl in PHPUnit:

     $mock = Mockery::mock(RollingCurl::class);
     $mock->shouldReceive('execute')->andReturnSelf();
     $mock->shouldReceive('get')->andReturnSelf();
     

   • Test edge cases: timeouts, redirects, and malformed responses.

4. Logging:

   • Log request/response metadata:

     $rollingCurl->setCallback(function($request) {
         Log::info("Fetched {$request->getUrl()}: {$request->getStatusCode()}");
     });
     

Gotchas and Tips

Pitfalls

1. Stale Codebase:

   • PHP 8.1+ Issues: The package may fail due to deprecated curl_multi_* functions. Workaround:

     // Polyfill for PHP 8.1+
     if (!function_exists('curl_multi_errno')) {
         function curl_multi_errno($mh) { /* ... */ }
     }
     

   • No Type Safety: Uses dynamic addOptions(), risking invalid curlopt values. Validate options:

     $validOptions = [CURLOPT_TIMEOUT, CURLOPT_HEADER, /* ... */];
     if (!in_array($option, $validOptions)) {
         throw new \InvalidArgumentException("Invalid cURL option");
     }
     

2. Memory Growth:

   • Unbounded Queue: Without prunePendingRequestQueue(), the pending request list grows indefinitely. Call it in the callback:

     $rollingCurl->setCallback(function($request, $rollingCurl) {
         $rollingCurl->prunePendingRequestQueue();
     });
     

   • Large Responses: Stream responses for big payloads (e.g., files):

     $request->addOptions([CURLOPT_WRITEFUNCTION => function($ch, $data) {
         file_put_contents('output.txt', $data, FILE_APPEND);
         return strlen($data);
     }]);
     

3. Callback Timing:

   • Callbacks fire asynchronously during execute(). Avoid relying on callback order for sequential logic.
   • For ordered processing, use a Synchronized wrapper or queue results to a database.

4. Connection Limits:

   • Default simultaneousLimit is unlimited if not set. Always configure it:

     $rollingCurl->setSimultaneousLimit(5); // Critical for production!
     

   • Overloading can trigger anti-DDoS measures (e.g., Cloudflare blocks).

5. Error Handling:

   • Silent Failures: curl_multi_* errors may not propagate. Check $request->getError() in the callback.
   • No Retries: Implement manually:

     $rollingCurl->setCallback(function($request) {
         if ($request->getError() && $attempts < 3) {
             $rollingCurl->add($request); // Requeue
         }
     });
     

6. Laravel-Specific Issues:

   • Service Container: The package isn’t PSR