Crawler Laravel Package

What's Changed

• Skip fnmatch for URLs longer than FILENAME_MAX by @mattiasgeniar in https://github.com/spatie/crawler/pull/505

Full Changelog: https://github.com/spatie/crawler/compare/9.3.0...9.3.1

What's Changed

• Add a shouldStopCallback hook for graceful external stops by @kissifrot in https://github.com/spatie/crawler/pull/504

Full Changelog: https://github.com/spatie/crawler/compare/9.2.1...9.3.0

What's Changed

• Close response body streams after processing by @freekmurze in https://github.com/spatie/crawler/pull/503

Full Changelog: https://github.com/spatie/crawler/compare/9.0.1...9.2.1

Full Changelog: https://github.com/spatie/crawler/compare/9.1.0...9.2.0

What's fixed

When the crawler reached its time limit, crawl limit, or was interrupted via a signal, any HTTP requests that were still in-flight in the Guzzle pool would eventually fail (e.g. due to their per-request timeout) and get reported to crawlFailed. These URLs were dispatched but never truly crawled, so reporting them as failures was misleading.

The CrawlRequestFailed handler now checks whether the crawler has reached its limits before reporting a failure. If it has, the failure is silently ignored.

Regular request failures (timeouts, connection errors, server errors) that happen during normal crawling are still reported as before.

What's Changed

• Modernize package - v9 by @freekmurze in https://github.com/spatie/crawler/pull/499

Full Changelog: https://github.com/spatie/crawler/compare/8.5.0...9.0.0

Add Laravel 13 support

What's Changed

• Update nicmart/tree dependency version to ^0.10 by @robinmiau in https://github.com/spatie/crawler/pull/497

New Contributors

• @robinmiau made their first contribution in https://github.com/spatie/crawler/pull/497

Full Changelog: https://github.com/spatie/crawler/compare/8.4.6...8.4.7

What's Changed

• When fetching robots.txt, use the same User-Agent as defined by the user by @mattiasgeniar in https://github.com/spatie/crawler/pull/491

Full Changelog: https://github.com/spatie/crawler/compare/8.4.4...8.4.5

What's Changed

• Update issue template by @AlexVanderbist in https://github.com/spatie/crawler/pull/488
• fix(CrawlUrl): Use before initialization is now impossible by @Voltra in https://github.com/spatie/crawler/pull/490

New Contributors

• @Voltra made their first contribution in https://github.com/spatie/crawler/pull/490

Full Changelog: https://github.com/spatie/crawler/compare/8.4.3...8.4.4

What's Changed

• Do not try robots.txt when ignored by @kissifrot in https://github.com/spatie/crawler/pull/485

New Contributors

• @kissifrot made their first contribution in https://github.com/spatie/crawler/pull/485

Full Changelog: https://github.com/spatie/crawler/compare/8.4.2...8.4.3

What's Changed

• set spatie/browsershot minimal version to 5.0.5 by @grafst in https://github.com/spatie/crawler/pull/484

New Contributors

• @grafst made their first contribution in https://github.com/spatie/crawler/pull/484

Full Changelog: https://github.com/spatie/crawler/compare/8.4.1...8.4.2

Full Changelog: https://github.com/spatie/crawler/compare/8.4.0...8.4.1

What's Changed

• Add execution time limit by @VincentLanglet in https://github.com/spatie/crawler/pull/480

New Contributors

• @VincentLanglet made their first contribution in https://github.com/spatie/crawler/pull/480

Full Changelog: https://github.com/spatie/crawler/compare/8.3.1...8.4.0

What's Changed

• Upgrade spatie/browsershot to 5.0 by @hasansoyalan in https://github.com/spatie/crawler/pull/478

New Contributors

• @hasansoyalan made their first contribution in https://github.com/spatie/crawler/pull/478

Full Changelog: https://github.com/spatie/crawler/compare/8.3.0...8.3.1

What's Changed

• Add support for PHP 8.4 by @pascalbaljet in https://github.com/spatie/crawler/pull/477

Full Changelog: https://github.com/spatie/crawler/compare/8.2.3...8.3.0

What's Changed

• Fix setParsableMimeTypes() by @superpenguin612 in https://github.com/spatie/crawler/pull/470

Full Changelog: https://github.com/spatie/crawler/compare/8.2.2...8.2.3

What's Changed

• Catch unhandled Browsershot exceptions in crawlFailed by @superpenguin612 in https://github.com/spatie/crawler/pull/469

Full Changelog: https://github.com/spatie/crawler/compare/8.2.1...8.2.2

What's Changed

• Check original URL against depth tree when visited link is a redirect by @superpenguin612 in https://github.com/spatie/crawler/pull/467

New Contributors

• @superpenguin612 made their first contribution in https://github.com/spatie/crawler/pull/467

Full Changelog: https://github.com/spatie/crawler/compare/8.2.0...8.2.1

What's Changed

• Fix wording in documentation by @adamtomat in https://github.com/spatie/crawler/pull/460
• Add Laravel/Illuminate 11 Support by @Jubeki in https://github.com/spatie/crawler/pull/461

New Contributors

• @adamtomat made their first contribution in https://github.com/spatie/crawler/pull/460
• @Jubeki made their first contribution in https://github.com/spatie/crawler/pull/461

Full Changelog: https://github.com/spatie/crawler/compare/8.1.0...8.2.0

What's Changed

• feat: custom link parser by @Velka-DEV in https://github.com/spatie/crawler/pull/458

New Contributors

• @Velka-DEV made their first contribution in https://github.com/spatie/crawler/pull/458

Full Changelog: https://github.com/spatie/crawler/compare/8.0.4...8.1.0

• allow Browsershot v4

What's Changed

• Fix return type by @riesjart in https://github.com/spatie/crawler/pull/452

New Contributors

• @riesjart made their first contribution in https://github.com/spatie/crawler/pull/452

Full Changelog: https://github.com/spatie/crawler/compare/8.0.2...8.0.3

What's Changed

• Define only needed methods in observer implementation by @buismaarten in https://github.com/spatie/crawler/pull/449

New Contributors

• @buismaarten made their first contribution in https://github.com/spatie/crawler/pull/449

Full Changelog: https://github.com/spatie/crawler/compare/8.0.1...8.0.2

What's Changed

• Check if rel attribute contains nofollow by @robbinbenard in https://github.com/spatie/crawler/pull/445

New Contributors

• @robbinbenard made their first contribution in https://github.com/spatie/crawler/pull/445

Full Changelog: https://github.com/spatie/crawler/compare/8.0.0...8.0.1

• add linkText to crawl observer methods
• upgrade dependencies

• support Laravel 10

What's Changed

• Feat/convert phpunit tests to pest by @mansoorkhan96 in https://github.com/spatie/crawler/pull/401
• Add the ability to change the default baseUrl scheme by @arnissolle in https://github.com/spatie/crawler/pull/402

New Contributors

• @arnissolle made their first contribution in https://github.com/spatie/crawler/pull/402

Full Changelog: https://github.com/spatie/crawler/compare/7.1.1...7.1.2

What's Changed

• Fix issue #395 by @BrokenSourceCode in https://github.com/spatie/crawler/pull/396

New Contributors

• @BrokenSourceCode made their first contribution in https://github.com/spatie/crawler/pull/396

Full Changelog: https://github.com/spatie/crawler/compare/7.1.0...7.1.1

• allow Laravel 9 collections

What's Changed

• Keep only guzzlehttp/psr7 v2.0 by @flangofas in https://github.com/spatie/crawler/pull/392

New Contributors

• @flangofas made their first contribution in https://github.com/spatie/crawler/pull/392

Full Changelog: https://github.com/spatie/crawler/compare/7.0.4...7.0.5

What's Changed

• Replace stream_for with Utils::streamFor by @NicolasMica in https://github.com/spatie/crawler/pull/391

New Contributors

• @NicolasMica made their first contribution in https://github.com/spatie/crawler/pull/391

Full Changelog: https://github.com/spatie/crawler/compare/7.0.3...7.0.4

• improve exception message

• allow psr7 v2

• change response type hint (#371)

• require PHP 8+
• drop support for PHP 7.x
• convert syntax to PHP 8
• no API changes have been made