Crawlerbundle Laravel Package

Getting Started

Minimal Setup

1. Installation

   composer require edemy/crawlerbundle
   

   Ensure eDemyFramework/EFrameworkBundle is installed as a dependency (this bundle is framework-specific).

2. Enable the Bundle Add to config/bundles.php:

   return [
       // ...
       Masando\CrawlerBundle\MasandoCrawlerBundle::class => ['all' => true],
   ];
   

3. Basic Configuration Publish the default config:

   php bin/console masando:crawler:install
   

   Edit config/packages/masando_crawler.yaml to define your first crawler (e.g., app.crawler.scraper).

4. First Crawl Define a crawler in YAML (e.g., config/packages/masando_crawler.yaml):

   masando_crawler:
       crawlers:
           app.crawler.scraper:
               url: "https://example.com"
               rules:
                   - selector: "h1"
                     callback: "App\Crawler\Callback\HeadlineCallback"
   

   Run via CLI:

   php bin/console masando:crawler:run app.crawler.scraper
   

5. Callback Class Create a callback handler (e.g., src/Crawler/Callback/HeadlineCallback.php):

   namespace App\Crawler\Callback;
   
   use Masando\CrawlerBundle\Callback\AbstractCallback;
   
   class HeadlineCallback extends AbstractCallback {
       public function handle($element) {
           return $element->textContent;
       }
   }
   

Implementation Patterns

Workflow: Scraping Dynamic Content

1. Queue Crawls Schedule crawls via Symfony’s Messenger component (configure in config/packages/masando_crawler.yaml):

   masando_crawler:
       crawlers:
           app.crawler.dynamic:
               url: "https://example.com/api/data"
               method: "POST"
               headers:
                   Accept: "application/json"
               rules:
                   - selector: "body"
                     callback: "App\Crawler\Callback\JsonCallback"
               schedule: "@hourly"
   

   Trigger via:

   php bin/console messenger:consume masando_crawler -vv
   

2. Paginate with Rules Use next_page_selector to crawl multi-page results:

   rules:
       - selector: ".product-item"
         callback: "App\Crawler\Callback\ProductCallback"
       - next_page_selector: ".pagination-next"
         url_pattern: "https://example.com/page/{page}"
   

3. Store Results Integrate with Doctrine or a queue (e.g., Redis) via custom callbacks:

   // Example: Save to database
   public function handle($element) {
       $entityManager = \Symfony\Bridge\Doctrine\ManagerRegistry::getManager();
       $product = new Product();
       $product->setName($element->querySelector('h2')->textContent);
       $entityManager->persist($product);
       $entityManager->flush();
       return $product;
   }
   

Integration Tips

1. Symfony Events Listen to masando.crawler.pre_crawl and masando.crawler.post_crawl events for pre/post-processing:

   // src/EventListener/CrawlerListener.php
   use Masando\CrawlerBundle\Event\CrawlerEvent;
   
   public function onPreCrawl(CrawlerEvent $event) {
       $event->getCrawler()->setOption('timeout', 30);
   }
   

2. Dependency Injection Inject the crawler service into controllers/services:

   use Masando\CrawlerBundle\Service\CrawlerService;
   
   public function __construct(private CrawlerService $crawler) {}
   
   public function scrape() {
       $result = $this->crawler->run('app.crawler.scraper');
       // Process $result
   }
   

3. Rate Limiting Configure delays between requests in YAML:

   masando_crawler:
       crawlers:
           app.crawler.limited:
               delay: 2 # seconds
               rules: [...]
   

Gotchas and Tips

Pitfalls

1. Selector Specificity

   • Issue: Fragile selectors (e.g., div#id) break if HTML changes.
   • Fix: Use more robust selectors (e.g., div.product > h2) or combine with attributes:

     selector: "div[class*='product-'] h2"
     

2. Callback Errors

   • Issue: Unhandled exceptions in callbacks crash the crawler.
   • Fix: Extend AbstractCallback and wrap logic in try-catch:

     public function handle($element) {
         try {
             return $this->processElement($element);
         } catch (\Exception $e) {
             $this->logger->error($e->getMessage());
             return null;
         }
     }
     

3. Dynamic URLs

   • Issue: Hardcoded URLs in url_pattern fail for dynamic routes.
   • Fix: Use placeholders ({page}) and pass context via context in YAML:

     url_pattern: "https://example.com/products?page={page}"
     context:
         page: 1
     

4. Memory Leaks

   • Issue: Large crawls exhaust memory.
   • Fix: Limit depth or use max_items in YAML:

     masando_crawler:
         crawlers:
             app.crawler.large:
                 max_items: 1000
     

Debugging

1. Log Output Enable verbose logging in config/packages/masando_crawler.yaml:

   masando_crawler:
       debug: true
   

   Check logs at var/log/dev.log.

2. Inspect HTML Use the dump_html option to save raw HTML for debugging:

   masando_crawler:
       crawlers:
           app.crawler.debug:
               dump_html: true
               output_dir: "%kernel.project_dir%/var/crawler_dumps"
   

3. Headless Testing Test crawlers locally with curl to mimic requests:

   curl -v -H "Accept: text/html" "https://example.com"
   

Extension Points

1. Custom Crawler Classes Extend Masando\CrawlerBundle\Crawler\AbstractCrawler for reusable logic:

   namespace App\Crawler;
   
   use Masando\CrawlerBundle\Crawler\AbstractCrawler;
   
   class ApiCrawler extends AbstractCrawler {
       protected function getClientOptions() {
           return [
               'headers' => ['Authorization' => 'Bearer token'],
           ];
       }
   }
   

   Register in YAML:

   masando_crawler:
       crawlers:
           app.crawler.api:
               class: "App\Crawler\ApiCrawler"
   

2. Middleware Add request/response middleware via masando_crawler.middleware tag:

   // src/Middleware/CustomMiddleware.php
   use Masando\CrawlerBundle\Middleware\MiddlewareInterface;
   
   class CustomMiddleware implements MiddlewareInterface {
       public function handle(\GuzzleHttp\Psr7\Request $request) {
           // Modify request
           return $request;
       }
   }
   

   Tag in services.yaml:

   services:
       App\Middleware\CustomMiddleware:
           tags:
               - { name: masando_crawler.middleware }
   

3. Post-Processing Chain callbacks or use masando.crawler.post_process event to transform results:

   // src/EventListener/PostProcessListener.php
   use Masando\CrawlerBundle\Event\PostProcessEvent;
   
   public function onPostProcess(PostProcessEvent $event) {
       $data = $event->getData();
       $event->setData(array_map(fn($item) => strtolower($item), $data));
   }