Laravel Sitemap Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require spatie/laravel-sitemap
   

   The package auto-registers.

2. First Crawl:

   use Spatie\Sitemap\SitemapGenerator;
   
   SitemapGenerator::create('https://example.com')
       ->writeToFile(public_path('sitemap.xml'));
   

   Run via CLI:

   php artisan sitemap:generate
   

3. First Manual Sitemap:

   use Spatie\Sitemap\Sitemap;
   use Spatie\Sitemap\Tags\Url;
   
   Sitemap::create()
       ->add(Url::create('/home'))
       ->writeToFile(public_path('sitemap.xml'));
   

Where to Look First

• Documentation: spatie.be/docs/laravel-sitemap
• Config: Publish with php artisan vendor:publish --provider="Spatie\Sitemap\SitemapServiceProvider" --tag=sitemap-config
• Crawler Profiles: Default in app/CrawlProfiles/ (create custom profiles here)

Implementation Patterns

1. Crawling + Manual Overrides

SitemapGenerator::create('https://example.com')
    ->getSitemap()
    ->add(Url::create('/static-page')->setLastModificationDate(now()))
    ->writeToFile(public_path('sitemap.xml'));

2. Model-Based Sitemaps (Sitemapable)

// Post.php
use Spatie\Sitemap\Contracts\Sitemapable;

class Post implements Sitemapable {
    public function toSitemapTag(): Url {
        return Url::create(route('posts.show', $this))
            ->setLastModificationDate($this->updated_at);
    }
}

// Generate via crawler
SitemapGenerator::create('https://example.com')
    ->crawlWithModel(Post::class)
    ->writeToFile(public_path('posts_sitemap.xml'));

3. Scheduled Generation

// app/Console/Commands/GenerateSitemap.php
SitemapGenerator::create(config('app.url'))
    ->writeToFile(public_path('sitemap.xml'));

// routes/console.php
Schedule::command('sitemap:generate')->daily();

4. Sitemap Index

use Spatie\Sitemap\SitemapIndex;

SitemapIndex::create()
    ->add('/posts_sitemap.xml')
    ->add('/pages_sitemap.xml')
    ->writeToFile(public_path('sitemap-index.xml'));

5. Dynamic URL Filtering

SitemapGenerator::create('https://example.com')
    ->hasCrawled(function (Url $url) {
        return ! str_contains($url->getAbsoluteUrl(), '/admin');
    })
    ->writeToFile(public_path('sitemap.xml'));

6. JavaScript-Rendered Pages

// config/sitemap.php
'execute_javascript' => true,
'chrome_binary_path' => '/path/to/chrome',

// Requires
composer require spatie/browsershot

Gotchas and Tips

Pitfalls

1. Crawl Depth Limits:

   • Default depth is 3. Increase with ->depth(5) if needed.
   • Fix: Configure in SitemapGenerator or via Crawler callback.

2. Robots.txt Overrides:

   • Crawler respects robots.txt by default. Disable with:

     ->configureCrawler(fn($crawler) => $crawler->ignoreRobots())
     

3. Memory Issues:

   • Large sites may hit memory limits during crawling.
   • Fix: Reduce concurrency (->setConcurrency(2)) or chunk URLs.

4. Dynamic Routes:

   • Crawler may miss JS-rendered routes. Use execute_javascript or manual Url additions.

5. Sitemap Size Limits:

   • Google recommends <50MB per sitemap. Split manually:

     Sitemap::create()->chunk(1000)->writeToFile(...);
     

Debugging Tips

• Log Crawled URLs:

  ->hasCrawled(function (Url $url) {
      \Log::info('Crawled:', [$url->getAbsoluteUrl()]);
      return $url;
  })
  

• Test Locally: Use ->writeToFile() before deploying to production.
• Validate Output: Use Google’s Sitemap Tester.

Extension Points

1. Custom Crawl Profiles: Extend Spatie\Sitemap\Crawler\Profile to modify crawl logic (e.g., exclude auth routes).

2. URL Tag Customization: Override Url tags via Sitemapable or hasCrawled callbacks:

   ->hasCrawled(function (Url $url) {
       if ($url->segment(1) === 'blog') {
           return $url->setChangeFrequency(Url::CHANGE_FREQUENCY_DAILY);
       }
       return $url;
   })
   

3. Event Listeners: Listen for sitemap.generated events to post-process sitemaps:

   // app/Providers/EventServiceProvider.php
   protected $listen = [
       'sitemap.generated' => [SitemapPostProcessor::class],
   ];
   

4. Headless Chrome:

   • Troubleshooting: Ensure Chrome is installed and accessible.
   • Performance: Disable if not needed (execute_javascript: false).

Config Quirks

• Guzzle Options: Customize HTTP client behavior in config/sitemap.php:

  'guzzle_options' => [
      'timeout' => 30,
      'headers' => ['User-Agent' => 'MyBot/1.0'],
  ],
  

• Disk Writing: Use writeToDisk('s3', 'sitemap.xml') for cloud storage (ensure visibility is set).

Performance Optimizations

• Cache Crawled URLs: Store results in Redis to avoid re-crawling:

  ->setCache(function () {
      return Cache::remember('sitemap_crawl', now()->addDays(1), fn() => []);
  })
  

• Parallel Generation: Use Laravel queues to generate sitemaps asynchronously.