Sitemapparser Laravel Package

Getting Started

composer require vipnytt/sitemapparser

The package is a thin, dependency-light parser: it fetches a URL, validates it against the Sitemaps.org XSD-ish rules, and exposes the contents as plain PHP arrays. There is one class you need, vipnytt\SitemapParser, and two methods that matter on day one.

use vipnytt\SitemapParser;

$parser = new SitemapParser('MyBot/1.0 (+https://example.com/bot)');
$parser->parse('https://example.com/sitemap.xml');

// For a <sitemapindex>: the child sitemaps
foreach ($parser->getSitemaps() as $loc => $meta) { /* ... */ }

// For a <urlset>: the actual pages
foreach ($parser->getURLs() as $loc => $meta) {
    // $meta['lastmod'], $meta['changefreq'], $meta['priority']
}

Start by looking at SitemapParser::parse() and the getURLs() / getSitemaps() return shapes — those two arrays are the entire public contract you will build on. Everything else (validation, gzip decoding, charset handling) is opaque to you.

The first realistic use case is a nightly crawl: point parse() at /sitemap.xml, check whether you got sitemaps or URLs back, and branch. That single call already handles .xml.gz sitemaps and redirects.

Note before you go further: the package is archived. Pin the version in composer.json (exact, not ^) and treat it as vendored code you now own.

Implementation Patterns

Walk sitemap indexes with your own queue. Treat getSitemaps() as a work list rather than relying on implicit recursion. A small BFS gives you depth limits, deduplication, and progress reporting for free:

$seen = [];
$queue = [['https://example.com/sitemap.xml', 0]];

while ($queue) {
    [$url, $depth] = array_shift($queue);
    if (isset($seen[$url]) || $depth > 3) continue;
    $seen[$url] = true;

    $parser->parse($url);

    foreach (array_keys($parser->getSitemaps()) as $child) {
        $queue[] = [$child, $depth + 1];
    }
    foreach ($parser->getURLs() as $loc => $meta) {
        $this->store($loc, $meta);
    }
}

Use non-strict mode for real-world crawling. Strict mode is the default and throws on the first malformed element, which is common in the wild. Construct with ['strict' => false], then drain getErrors() after each parse() and log them. You get partial results plus a diagnostic trail instead of an aborted run.

Normalise at the boundary. The parser gives you raw strings. Convert lastmod to DateTimeImmutable yourself (new DateTimeImmutable($meta['lastmod']) wrapped in a try/catch — timezone-less and oddly formatted values are routine), and cast priority to float. Do this once in a DTO so downstream jobs never see the parser's arrays.

Persist by loc, not by row id. Hash the URL for the primary key and store lastmod; on re-crawl, skip rows whose lastmod has not changed. This is what makes weekly sitemap polling cheap.

Wrap it behind an interface. Because the package is archived, define your own SitemapSource contract with a urls(string $sitemapUrl): iterable method and make the library one implementation. When you eventually swap to a hand-rolled XMLReader loop, only one class changes. This also gives you a seam for tests.

Fake it in tests via the Guzzle options. Pass ['guzzle' => ['handler' => $mockHandler]] (or equivalent transport options) in the constructor so parse() never touches the network. Fixture files covering a nested index, a gzipped sitemap, and a malformed entry are the three cases worth committing.

Run it as a queued job per sitemap. One job per sitemap document, dispatched from a parent job that owns the queue/dedupe state, keeps memory flat and lets you retry individual documents with backoff.

Gotchas and Tips

• Archived means frozen. No fixes for new PHP versions, no security patches, no upstream response to changed sitemap conventions. Pin the exact version and consider copying the class into your namespace if you need to patch it.
• Strict mode is the default. If your first run throws on a sitemap you can open fine in a browser, that is why. Set 'strict' => false and inspect getErrors().
• Results are keyed by loc, so duplicates collapse. Two entries with the same URL silently become one. If you need per-entry fidelity, you cannot get it from getURLs().
• No cycle protection in sitemap indexes. A sitemap that lists itself, or an A→B→A pair, will loop forever if you trust recursion. Always keep a visited set and a depth cap.
• Relative and protocol-relative URLs are not resolved. You get the string as written in <loc>. Run everything through a URL resolver before you fetch it, or you will issue requests to nonsense.
• Set a real user agent. Many servers 403 a blank or default UA. The constructor's first argument is not cosmetic.
• Memory scales with the document, not with your loop. A 50k-URL sitemap becomes a large array. Set a size/time limit in the transport options, and prefer more, smaller sitemaps on the generating side.
• parse() in strict mode throws; in non-strict it returns quietly with errors queued. Wrap calls in try/catch regardless if you might flip modes later, and clear/read getErrors() per document so messages do not accumulate across a long crawl.
• Namespaces and non-standard XML are the usual failure point. Sitemaps that omit the namespace, mix in <url> elements at index level, or use a custom extension namespace will parse poorly or not at all. Keep an eye on the error log for these and skip the document rather than retrying.
• Respect robots.txt separately. This package parses sitemaps, not crawl permissions; you still need to check Sitemap: directives and Disallow rules yourself before fetching.
• The extension point is the transport layer. Any custom auth headers, proxies, or retry policy belong in the constructor options, not in a subclass — there is no meaningful hook to override in the parsing logic itself.