Laravel Sitemap Laravel Package

Getting Started

Minimal Steps

1. Installation:

   composer require spatie/laravel-sitemap
   

   The package auto-registers.

2. Basic Crawl & Generate:

   use Spatie\Sitemap\SitemapGenerator;
   
   SitemapGenerator::create('https://example.com')
       ->writeToFile(public_path('sitemap.xml'));
   

3. First Use Case:

   • Run the crawler once to generate a sitemap for your live site.
   • Verify the output at public/sitemap.xml and validate it using Google's Sitemap Tester.

Implementation Patterns

Core Workflows

1. Crawl-Based Generation:

   SitemapGenerator::create('https://example.com')
       ->setMaximumCrawlCount(1000) // Limit depth
       ->setConcurrency(5)          // Parallel requests
       ->writeToFile(public_path('sitemap.xml'));
   

2. Hybrid Approach (Crawl + Manual URLs):

   $generator = SitemapGenerator::create('https://example.com');
   $sitemap = $generator->getSitemap();
   
   // Add manual URLs (e.g., admin pages)
   $sitemap->add(Url::create('/admin')->setChangeFrequency(Url::CHANGE_FREQUENCY_NEVER));
   
   $sitemap->writeToFile(public_path('sitemap.xml'));
   

3. Model-Based Sitemaps (via Sitemapable):

   // In Post.php
   class Post implements Sitemapable {
       public function toSitemapTag(): Url {
           return Url::create(route('posts.show', $this))
               ->setLastModificationDate($this->updated_at)
               ->setChangeFrequency(Url::CHANGE_FREQUENCY_WEEKLY);
       }
   }
   
   // Generate via:
   SitemapGenerator::create('https://example.com')
       ->getSitemap()
       ->add(Post::all()->map(fn($post) => $post->toSitemapTag()))
       ->writeToFile(public_path('posts_sitemap.xml'));
   

4. Scheduled Generation (via Artisan Command):

   // app/Console/Commands/GenerateSitemap.php
   class GenerateSitemap extends Command {
       public function handle() {
           SitemapGenerator::create(config('app.url'))
               ->writeToFile(public_path('sitemap.xml'));
       }
   }
   
   // Schedule in routes/console.php:
   Schedule::command('sitemap:generate')->dailyAt('2:00');
   

Integration Tips

1. Dynamic URL Filtering:

   SitemapGenerator::create('https://example.com')
       ->hasCrawled(fn(Url $url) => !Str::contains($url->getAbsoluteUrl(), ['/login', '/admin']))
       ->writeToFile(public_path('sitemap.xml'));
   

2. Multilingual Sitemaps:

   $sitemap = Sitemap::create();
   $sitemap->add(
       Url::create('/post')
           ->addAlternate('/post-fr', 'fr')
           ->addAlternate('/post-es', 'es')
   );
   

3. News Sitemaps:

   $sitemap->add(
       Url::create('/news/breaking')
           ->addNews('My News Site', 'en', 'Breaking News', now())
   );
   

4. Sitemap Indexing (for large sites):

   SitemapIndex::create()
       ->add('/posts_sitemap.xml')
       ->add('/pages_sitemap.xml')
       ->writeToDisk('public', 'sitemap-index.xml');
   

5. Headless Chrome for JS-Rendered Pages:

   // config/sitemap.php
   'execute_javascript' => true,
   'chrome_binary_path' => '/path/to/chrome',
   
   // Requires:
   composer require spatie/browsershot
   

Gotchas and Tips

Pitfalls

1. Crawl Depth Limits:

   • Default depth is unlimited, but may cause performance issues.
   • Fix: Use ->depth(3) or ->setMaximumCrawlCount(500) to constrain crawling.

2. Robots.txt Overrides:

   • The crawler respects robots.txt by default.
   • Fix: Disable with ->ignoreRobots() if needed (e.g., for internal pages).

3. Concurrency Issues:

   • High concurrency (default: 10) may trigger rate-limiting.
   • Fix: Reduce with ->setConcurrency(2) for sensitive APIs.

4. Dynamic Content:

   • Crawler may miss JS-rendered links.
   • Fix: Enable execute_javascript (requires spatie/browsershot).

5. Duplicate URLs:

   • Crawler may add the same URL multiple times.
   • Fix: Use ->hasCrawled() to deduplicate:

     ->hasCrawled(fn(Url $url) => !collect($crawledUrls)->contains($url->getAbsoluteUrl()))
     

6. File Permissions:

   • Writing to public/ may fail if permissions are restrictive.
   • Fix: Ensure storage:link is run and public/ is writable.

Debugging Tips

1. Log Crawled URLs:

   SitemapGenerator::create('https://example.com')
       ->hasCrawled(fn(Url $url) => {
           Log::debug('Crawled:', [$url->getAbsoluteUrl()]);
           return $url;
       })
       ->writeToFile(...);
   

2. Validate Sitemap:

   • Use Google’s Sitemap Tester.
   • Check for:
     • Malformed URLs.
     • Missing lastmod dates.
     • Duplicate entries.

3. Profile Crawl Performance:

   $start = microtime(true);
   SitemapGenerator::create('https://example.com')->writeToFile(...);
   Log::info('Sitemap generated in ' . (microtime(true) - $start) . 's');
   

Extension Points

1. Custom Crawl Profiles: Extend Spatie\Sitemap\Crawler\Profile to modify crawl behavior:

   class CustomProfile extends Profile {
       public function shouldCrawl(string $url): bool {
           return !Str::contains($url, ['/private']);
       }
   }
   
   // Usage:
   SitemapGenerator::create('https://example.com')
       ->setCrawlProfile(new CustomProfile());
   

2. Custom URL Tags: Extend Spatie\Sitemap\Tags\Url for custom XML attributes:

   class CustomUrl extends Url {
       public function addCustomAttribute(string $name, string $value): self {
           $this->customAttributes[$name] = $value;
           return $this;
       }
   }
   

3. Pre/Post-Crawl Hooks: Use Laravel events to hook into the process:

   // In EventServiceProvider:
   protected $listen = [
       'sitemap.generating' => [SitemapHook::class, 'onGenerating'],
       'sitemap.generated'  => [SitemapHook::class, 'onGenerated'],
   ];
   

4. Cache Warmup: Pre-generate sitemaps during deployments:

   // In Deployer task:
   task('deploy:sitemap', function() {
       run('php artisan sitemap:generate');
   });
   

Configuration Quirks

1. guzzle_options:

   • Overrides default Guzzle settings (e.g., timeouts, headers).
   • Example:

     'guzzle_options' => [
         'timeout' => 30,
         'headers' => ['User-Agent' => 'MySitemapBot/1.0'],
     ],
     

2. Chrome Binary Path:

   • If execute_javascript is true, ensure chrome_binary_path is set correctly.
   • Tip: Use which chrome (Linux/macOS) or where chrome (Windows) to locate the binary.

3. Disk Writing:

   • writeToDisk() uses Laravel’s filesystem. Ensure the disk is configured in config/filesystems.php.
   • Example:

     ->writeToDisk('s3', 'sitemap.xml', true) // Public visibility