Content Parser Laravel Package

Getting Started

Minimal Setup

1. Installation

   composer require deepslam/content-parser
   

   Add to composer.json if not auto-discovered:

   "autoload": {
       "psr-4": {
           "App\\": "app/",
           "Deepslam\\ContentParser\\": "vendor/deepslam/content-parser/src/"
       }
   }
   

   Run composer dump-autoload.

2. First Use Case Parse a URL’s main content:

   use Deepslam\ContentParser\ContentParser;
   
   $parser = new ContentParser();
   $content = $parser->parse('https://example.com');
   echo $content->getContent(); // Returns the extracted text
   

3. Key Classes to Explore

   • ContentParser: Main class for parsing.
   • Content: Holds parsed results (title, content, images, etc.).
   • Parser\*: Concrete parsers (e.g., Parser\DefaultParser, Parser\MediumParser).

Implementation Patterns

Workflows

1. Basic Parsing

   $parser = new ContentParser();
   $content = $parser->parse('https://blog.example.com/post');
   $title = $content->getTitle();
   $text = $content->getContent();
   

2. Custom Parser Selection Override default parser for specific domains:

   $parser = new ContentParser();
   $parser->setParser('medium', new \Deepslam\ContentParser\Parser\MediumParser());
   $content = $parser->parse('https://medium.com/...');
   

3. Batch Processing Useful for scraping multiple pages (e.g., RSS feeds or sitemaps):

   $urls = ['url1', 'url2', 'url3'];
   $results = [];
   foreach ($urls as $url) {
       $results[$url] = (new ContentParser())->parse($url);
   }
   

4. Integration with Laravel

   • Service Provider:

     // app/Providers/ContentParserServiceProvider.php
     public function register() {
         $this->app->singleton(ContentParser::class, function ($app) {
             return new ContentParser();
         });
     }
     

   • Facade (Optional): Create a facade for cleaner syntax:

     // app/Facades/ContentParserFacade.php
     public static function parse($url) {
         return app(ContentParser::class)->parse($url);
     }
     

5. Storing Results Save parsed content to a database (e.g., using Eloquent):

   $parsed = (new ContentParser())->parse('https://example.com');
   $post = new Post();
   $post->title = $parsed->getTitle();
   $post->content = $parsed->getContent();
   $post->save();
   

Gotchas and Tips

Pitfalls

1. Outdated Dependencies

   • The package was last updated in 2017 and may rely on deprecated libraries (e.g., guzzlehttp/guzzle: ~5.0).
   • Mitigation: Use a wrapper or fork to update dependencies (e.g., Guzzle 6/7).

2. Fragile Parsing Logic

   • Parsers use regex and DOM traversal, which break easily with layout changes.
   • Tip: Test on a variety of sites. Fall back to a generic parser if specific ones fail:

     try {
         $content = $parser->parse($url);
     } catch (\Exception $e) {
         $parser->setParser('default', new \Deepslam\ContentParser\Parser\DefaultParser());
         $content = $parser->parse($url);
     }
     

3. Rate Limiting

   • Aggressive scraping may trigger blocks. Add delays:

     sleep(1); // 1-second delay between requests
     

4. Encoding Issues

   • Some sites use non-UTF-8 encodings (e.g., ISO-8859-1). Normalize output:

     $content = mb_convert_encoding($content, 'UTF-8', 'auto');
     

Debugging

1. Inspect Parsed HTML Dump the raw HTML before parsing to debug selectors:

   $html = file_get_contents('https://example.com');
   // Manually inspect $html or use `var_dump($parser->getDom())`.
   

2. Log Failures Track URLs that fail parsing:

   try {
       $content = $parser->parse($url);
   } catch (\Exception $e) {
       Log::error("Failed to parse $url: " . $e->getMessage());
   }
   

Extension Points

1. Custom Parsers Extend \Deepslam\ContentParser\Parser\AbstractParser to handle niche sites:

   class CustomParser extends AbstractParser {
       protected function parse() {
           // Custom logic (e.g., XPath queries)
           return $this->extractContent();
       }
   }
   

2. Pre/Post-Processing Hook into the pipeline:

   $parser = new ContentParser();
   $parser->setPreParser(function ($html) {
       // Modify HTML before parsing (e.g., remove ads)
       return $html;
   });
   $parser->setPostParser(function ($content) {
       // Clean up text (e.g., remove extra whitespace)
       return trim($content);
   });
   

3. Caching Cache parsed results to avoid repeated requests:

   $cacheKey = 'parsed_content_' . md5($url);
   if (Cache::has($cacheKey)) {
       $content = Cache::get($cacheKey);
   } else {
       $content = (new ContentParser())->parse($url);
       Cache::put($cacheKey, $content, now()->addHours(1));
   }
   

Config Quirks

• No Built-in Config: The package is lightweight and doesn’t require configuration.
• User-Agent Spoofing: Add headers to mimic a browser:

  $parser->setClient(new \GuzzleHttp\Client([
      'headers' => ['User-Agent' => 'Mozilla/5.0']
  ]));