Laravel Sitemap Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require spatie/laravel-sitemap
   

   The package auto-registers.

2. First Crawl:

   use Spatie\Sitemap\SitemapGenerator;
   
   SitemapGenerator::create('https://your-site.com')
       ->writeToFile(public_path('sitemap.xml'));
   

3. First Manual Sitemap:

   use Spatie\Sitemap\Sitemap;
   use Spatie\Sitemap\Tags\Url;
   
   Sitemap::create()
       ->add(Url::create('/home'))
       ->writeToFile(public_path('sitemap.xml'));
   

Where to Look First

• Documentation: spatie.be/docs/laravel-sitemap
• Config: Publish with php artisan vendor:publish --provider="Spatie\Sitemap\SitemapServiceProvider" --tag=sitemap-config
• Crawler Docs: spatie/crawler

Implementation Patterns

1. Crawling + Manual Overrides

SitemapGenerator::create('https://your-site.com')
    ->getSitemap()
    ->add(Url::create('/special-page')->setLastModificationDate(Carbon::yesterday()))
    ->writeToFile(public_path('sitemap.xml'));

2. Model-Based Sitemaps (Sitemapable)

// In your model (e.g., Post.php)
use Spatie\Sitemap\Contracts\Sitemapable;

class Post implements Sitemapable {
    public function toSitemapTag(): Url {
        return Url::create(route('posts.show', $this))
            ->setLastModificationDate($this->updated_at);
    }
}

// Generate via crawler or manually
SitemapGenerator::create('https://your-site.com')
    ->getSitemap()
    ->add(Url::createFromModel($post))
    ->writeToFile(public_path('sitemap.xml'));

3. Scheduled Generation

// app/Console/Commands/GenerateSitemap.php
SitemapGenerator::create(config('app.url'))
    ->writeToFile(public_path('sitemap.xml'));

// routes/console.php
Schedule::command('sitemap:generate')->daily();

4. Sitemap Indexing

use Spatie\Sitemap\SitemapIndex;

SitemapIndex::create()
    ->add('/posts_sitemap.xml')
    ->add('/pages_sitemap.xml')
    ->writeToDisk('public', 'sitemap-index.xml');

5. Dynamic URL Filtering

SitemapGenerator::create('https://your-site.com')
    ->hasCrawled(function (Url $url) {
        return ! str_contains($url->getAbsoluteUrl(), '/admin');
    })
    ->writeToFile(public_path('sitemap.xml'));

6. Writing to Disks (S3, etc.)

SitemapGenerator::create('https://your-site.com')
    ->getSitemap()
    ->writeToDisk('s3', 'sitemap.xml', true); // Public visibility

Gotchas and Tips

Pitfalls

1. Crawl Depth Limits:

   • Default depth is null (unlimited). Set explicitly to avoid infinite loops:

     ->configureCrawler(fn($crawler) => $crawler->depth(3))
     

2. JavaScript Execution Overhead:

   • Requires spatie/browsershot and Chrome. Disable if unused (execute_javascript: false in config).
   • Chrome binary path must be set if not auto-detected.

3. Robots.txt Respect:

   • Crawler respects robots.txt by default. Use ignoreRobots() to bypass:

     ->configureCrawler(fn($crawler) => $crawler->ignoreRobots())
     

4. Concurrency Issues:

   • Default concurrency is 10. Reduce for rate-limited sites:

     ->setConcurrency(2)
     

5. URL Duplication:

   • Crawler may add duplicate URLs. Use hasCrawled to deduplicate:

     ->hasCrawled(fn(Url $url) => ! collect($existingUrls)->contains($url->getAbsoluteUrl()))
     

6. Model Changes Not Reflected:

   • Manually regenerate sitemaps after model updates or use scheduled tasks.

Debugging Tips

1. Log Crawled URLs:

   ->hasCrawled(function (Url $url) {
       \Log::info('Crawled:', ['url' => $url->getAbsoluteUrl()]);
       return $url;
   })
   

2. Check Crawler Profile:

   • Extend Spatie\Sitemap\Crawler\Profile to customize behavior (e.g., shouldCrawl logic).

3. Validate XML Output:

   • Use online tools like XML Validation to verify sitemap syntax.

4. Test Locally:

   • Mock HTTP requests with GuzzleHttp\HandlerStack for offline testing:

     $stack = HandlerStack::create();
     $stack->push(Middleware::tap(fn($request) => \Log::info('Request:', $request->getUri())));
     SitemapGenerator::create('https://your-site.com')->setGuzzleClient(new Client(['handler' => $stack]));
     

Extension Points

1. Custom Crawl Profiles:

   // app/CustomProfile.php
   use Spatie\Sitemap\Crawler\Profile;
   
   class CustomProfile extends Profile {
       public function shouldCrawl(string $url): bool {
           return ! str_contains($url, '/private');
       }
   }
   
   // In config/sitemap.php
   'crawl_profile' => \App\CustomProfile::class,
   

2. Custom URL Tags:

   • Extend Spatie\Sitemap\Tags\Url or create new tags (e.g., for video sitemaps).

3. Pre/Post-Crawl Hooks:

   SitemapGenerator::create('https://your-site.com')
       ->beforeCrawl(fn() => \Log::info('Starting crawl...'))
       ->afterCrawl(fn() => \Log::info('Crawl complete'))
       ->writeToFile(public_path('sitemap.xml'));
   

4. Dynamic Sitemap Routes:

   // routes/web.php
   Route::get('/sitemap.xml', function() {
       return SitemapGenerator::create('https://your-site.com')
           ->getSitemap()
           ->toXml();
   });
   

Performance Optimizations

1. Cache Crawled URLs:

   ->setCachePath(storage_path('crawled_urls.cache'))
   

2. Limit Crawl Count:

   ->setMaximumCrawlCount(1000) // Avoid over-crawling
   

3. Use Disk Storage:

   • Write sitemaps to disk instead of memory for large sites:

     ->writeToDisk('public', 'sitemap.xml')
     

4. Parallelize Sitemap Generation:

   • Generate multiple sitemaps (e.g., /posts, /pages) in parallel using Laravel queues.