Crawler Laravel Package

What's Changed

• Close response body streams after processing by @freekmurze in https://github.com/spatie/crawler/pull/503

Full Changelog: https://github.com/spatie/crawler/compare/9.0.1...9.2.1

Full Changelog: https://github.com/spatie/crawler/compare/9.1.0...9.2.0

Fixed

• Fixed allow_redirects default: changed from false to ['track_redirects' => true] so redirects are followed and the redirect history header is populated correctly
• Non-parseable responses (e.g. binary files filtered by allowedMimeTypes) now notify observers via crawled() with an empty body instead of being silently skipped
• URLs containing control characters are now detected and reported as malformed
• Dot segments (/../, /./) in extracted URLs are now normalized per RFC 3986
• Custom client options passed to Crawler::create() now merge with defaults instead of replacing them (pass null to remove a default)
• CrawlRequestFailed now wraps non-RequestException errors so observers always receive a RequestException
• Removed unused classes: Url, ResponseWithCachedBody, InvalidUrl

Added

• stream() method to opt-in to streaming HTTP responses for reduced memory usage
• matchWww() method to treat www.example.com and example.com as equivalent when using internalOnly()
• includeSubdomains() now works as a flag on internalOnly() and composes with matchWww()
• CrawlResponse::redirectHistory() and CrawlResponse::wasRedirected() for inspecting redirect chains
• CrawlObserver::crawlFailed() now receives a ?TransferStatistics parameter for detecting timeouts
• addObserver() now accepts variadic arguments: addObserver($obs1, $obs2)
• CrawlRequestFailed now preserves the original request from ConnectException (retaining custom headers like X-Started-At)

Major rewrite. See UPGRADING.md for a full list of breaking changes.

Changed

• Replace UriInterface with plain string URLs throughout the API
• Replace ResponseInterface with CrawlResponse in observer callbacks
• CrawlProfile is now an interface instead of an abstract class
• CrawlObserverCollection no longer implements ArrayAccess or Iterator
• Default scheme changed from http to https
• JavaScript rendering is now driver-based (Browsershot moved to suggest)
• UrlParser interface redesigned to return ExtractedUrl[] instead of adding to queue directly
• CrawlQueue::has() now accepts string instead of CrawlUrl|UriInterface
• start() now returns a FinishReason enum
• URL is now required in Crawler::create()

Added

• CrawlResponse object with status(), body(), dom(), header(), transferStats(), and more
• CrawlProgress tracking with urlsCrawled, urlsFailed, urlsFound, urlsPending
• FinishReason enum: Completed, CrawlLimitReached, TimeLimitReached, Interrupted
• Closure callbacks: onCrawled(), onFailed(), onFinished(), onWillCrawl()
• foundUrls() to collect all URLs as CrawledUrl objects
• fake() for testing without HTTP requests
• Scope helpers: internalOnly(), includeSubdomains(), shouldCrawl()
• Shorter method names: depth(), concurrency(), delay(), limit(), userAgent()
• Throttling: FixedDelayThrottle and AdaptiveThrottle
• Resource type extraction: alsoExtract(), extractAll(), ResourceType enum
• URL normalization in ArrayCrawlQueue
• Graceful shutdown via SIGINT/SIGTERM
• alwaysCrawl() and neverCrawl() pattern overrides
• retry() for automatic retries on connection errors and 5xx responses
• TransferStatistics with typed timing accessors
• CloudflareRenderer for JavaScript rendering
• JavaScriptRenderer interface for custom renderers
• Request configuration: basicAuth(), token(), withoutVerifying(), proxy(), cookies(), queryParameters(), middleware()

Removed

• CrawlUrl::create() static factory (use new CrawlUrl(...) instead)
• Spatie\Crawler\Url class
• ResponseWithCachedBody (replaced by CrawlResponse)
• nicmart/tree dependency
• spatie/browsershot as a required dependency (moved to suggest)
• setBrowsershot() and getBrowsershot() methods
• startCrawling() method (use start())
• setUrlParserClass() (use parseSitemaps() or pass a UrlParser directly)

Add Laravel 13 support

What's Changed

• Update nicmart/tree dependency version to ^0.10 by @robinmiau in https://github.com/spatie/crawler/pull/497

New Contributors

• @robinmiau made their first contribution in https://github.com/spatie/crawler/pull/497

Full Changelog: https://github.com/spatie/crawler/compare/8.4.6...8.4.7

What's Changed

• When fetching robots.txt, use the same User-Agent as defined by the user by @mattiasgeniar in https://github.com/spatie/crawler/pull/491

Full Changelog: https://github.com/spatie/crawler/compare/8.4.4...8.4.5

What's Changed

• Update issue template by @AlexVanderbist in https://github.com/spatie/crawler/pull/488
• fix(CrawlUrl): Use before initialization is now impossible by @Voltra in https://github.com/spatie/crawler/pull/490

New Contributors

• @Voltra made their first contribution in https://github.com/spatie/crawler/pull/490

Full Changelog: https://github.com/spatie/crawler/compare/8.4.3...8.4.4

What's Changed

• Do not try robots.txt when ignored by @kissifrot in https://github.com/spatie/crawler/pull/485

New Contributors

• @kissifrot made their first contribution in https://github.com/spatie/crawler/pull/485

Full Changelog: https://github.com/spatie/crawler/compare/8.4.2...8.4.3

What's Changed

• set spatie/browsershot minimal version to 5.0.5 by @grafst in https://github.com/spatie/crawler/pull/484

New Contributors

• @grafst made their first contribution in https://github.com/spatie/crawler/pull/484

Full Changelog: https://github.com/spatie/crawler/compare/8.4.1...8.4.2

Full Changelog: https://github.com/spatie/crawler/compare/8.4.0...8.4.1

What's Changed

• Add execution time limit by @VincentLanglet in https://github.com/spatie/crawler/pull/480

New Contributors

• @VincentLanglet made their first contribution in https://github.com/spatie/crawler/pull/480

Full Changelog: https://github.com/spatie/crawler/compare/8.3.1...8.4.0

What's Changed

• Upgrade spatie/browsershot to 5.0 by @hasansoyalan in https://github.com/spatie/crawler/pull/478

New Contributors

• @hasansoyalan made their first contribution in https://github.com/spatie/crawler/pull/478

Full Changelog: https://github.com/spatie/crawler/compare/8.3.0...8.3.1

What's Changed

• Add support for PHP 8.4 by @pascalbaljet in https://github.com/spatie/crawler/pull/477

Full Changelog: https://github.com/spatie/crawler/compare/8.2.3...8.3.0

What's Changed

• Fix setParsableMimeTypes() by @superpenguin612 in https://github.com/spatie/crawler/pull/470

Full Changelog: https://github.com/spatie/crawler/compare/8.2.2...8.2.3

What's Changed

• Check original URL against depth tree when visited link is a redirect by @superpenguin612 in https://github.com/spatie/crawler/pull/467

New Contributors

• @superpenguin612 made their first contribution in https://github.com/spatie/crawler/pull/467

Full Changelog: https://github.com/spatie/crawler/compare/8.2.0...8.2.1

What's Changed

• Fix wording in documentation by @adamtomat in https://github.com/spatie/crawler/pull/460
• Add Laravel/Illuminate 11 Support by @Jubeki in https://github.com/spatie/crawler/pull/461

New Contributors

• @adamtomat made their first contribution in https://github.com/spatie/crawler/pull/460
• @Jubeki made their first contribution in https://github.com/spatie/crawler/pull/461

Full Changelog: https://github.com/spatie/crawler/compare/8.1.0...8.2.0

What's Changed

• feat: custom link parser by @Velka-DEV in https://github.com/spatie/crawler/pull/458

New Contributors

• @Velka-DEV made their first contribution in https://github.com/spatie/crawler/pull/458

Full Changelog: https://github.com/spatie/crawler/compare/8.0.4...8.1.0

• allow Browsershot v4

What's Changed

• Fix return type by @riesjart in https://github.com/spatie/crawler/pull/452

New Contributors

• @riesjart made their first contribution in https://github.com/spatie/crawler/pull/452

Full Changelog: https://github.com/spatie/crawler/compare/8.0.2...8.0.3

What's Changed

• Define only needed methods in observer implementation by @buismaarten in https://github.com/spatie/crawler/pull/449

New Contributors

• @buismaarten made their first contribution in https://github.com/spatie/crawler/pull/449

Full Changelog: https://github.com/spatie/crawler/compare/8.0.1...8.0.2

What's Changed

• Check if rel attribute contains nofollow by @robbinbenard in https://github.com/spatie/crawler/pull/445

New Contributors

• @robbinbenard made their first contribution in https://github.com/spatie/crawler/pull/445

Full Changelog: https://github.com/spatie/crawler/compare/8.0.0...8.0.1

• add linkText to crawl observer methods
• upgrade dependencies

• support Laravel 10

What's Changed

• Feat/convert phpunit tests to pest by @mansoorkhan96 in https://github.com/spatie/crawler/pull/401
• Add the ability to change the default baseUrl scheme by @arnissolle in https://github.com/spatie/crawler/pull/402

New Contributors

• @arnissolle made their first contribution in https://github.com/spatie/crawler/pull/402

Full Changelog: https://github.com/spatie/crawler/compare/7.1.1...7.1.2

• allow Laravel 9 collections

• allow psr7 v2

• change response type hint (#371)

• require PHP 8+
• drop support for PHP 7.x
• convert syntax to PHP 8
• no API changes have been made

• bugfix: infinite loops when a CrawlProfile prevents crawling (#358)

• add setCurrentCrawlLimit and setTotalCrawlLimit
• internal refactors

• add support for PHP 8.0

• tweak variable naming in ArrayCrawlQueue (#326)

• improve chucked reading of response
• move observer / profiles / queues to separate namespaces
• typehint all the things
• use laravel/collections instead of tightenco package
• remove support for anything below PHP 7.4
• remove all deprecated functions and classes

• treat connection exceptions as request exceptions

• fix: method and property name error (#311)

• add crawler option to allow crawl links with rel="nofollow" (#310)

• only crawl links that are completely parsed

• fix curl streaming responses (#295)

• add setParseableMimeTypes() (#293)

• fix LinkAdder not receiving the updated DOM (#292)

• allow tightenco/collect 7 (#282)

• respect maximum response size when checking Robots Meta tags (#281)

• allow Guzzle 7

• allow symfony 5 components

• allow tightenco/collect 6.0 and up (#261)

• fix crash when CrawlRequestFailed receives an exception other than RequestException

• case-insensitive user agent bugfix (#249)

• fix bugs in hasAlreadyBeenProcessed

THIS VERSION CONTAINS A CRITICAL BUG, DO NOT USE

• added ArrayCrawlQueue; this is now the default queue
• deprecated CollectionCrawlQueue

• Make user agent configurable (#246)

• delayBetweenRequests now uses int instead of float everywhere

• remove incorrect docblock

• handle relative paths after redirects correctly

• add getUrls and getPendingUrls

• Respect maximumDepth in combination with robots (#181)

• Properly handle noindex,follow urls.

• added capability of crawling links with rel= next or prev

• add setDelayBetweenRequests

• fix an issue where the node in the depthtree could be null

• improve performance by only building the depth three when needed
• handlers will get html after JavaScript has been processed

• refactor to improve extendability

• always add links to pool if robots shouldn't be respected

• refactor of internals

• make it possible to override $defaultClientOptions

• Bump minimum required version of spatie/robots-txt to 1.0.1.

• Respect robots.txt

• improved extensibility by removing php native type hinting of url, queue and crawler pool Closures

• do not follow links that have attribute rel set to nofollow

• Support both Illuminate's and Tighten's Collection.

• fix bugs when installing into a Laravel app

• the CrawlObserver and CrawlProfile are upgraded from interfaces to abstract classes
• don't crawl tel: links

• fix endless loop

• add setCrawlObservers, addCrawlObserver

• fix setMaximumResponseSize (someday we'll get this right)

CONTAINS BUGS, DO NOT USE THIS VERSION

• fix setMaximumResponseSize

CONTAINS BUGS, DO NOT USE THIS VERSION

• fix setMaximumResponseSize

CONTAINS BUGS, DO NOT USE THIS VERSION

• add setMaximumResponseSize

• fix for exception being thrown when encountering a malformatted url

• use \Psr\Http\Message\UriInterface for all urls
• use Puppeteer
• drop support from PHP 7.0

• allow symfony 4 crawler

• added the ability to change the crawl queue

• more performance improvements

• performance improvements

• add CrawlSubdomains profile

• add crawl count limit

• add depth limit

• add JavaScript execution

• fix deps for PHP 7.2

• add EmptyCrawlObserver

• refactor to make use of Symfony Crawler's link function

• fix bugs around relative urls

• add CrawlInternalUrls

• make sure the passed client options are being used

• second attempt to fix detection of redirects

• fix detection of redirects

• fix the default timeout of 5 seconds

• set a default timeout of 5 seconds

• fix for non responding hosts

• fix for the accidental crawling of mailto-links

• improve performance by concurrent crawling
• make it possible to determine on which url a url was found

• Add support for DomCrawler 3.x

• Fix for normalizing relative links when using non-80 ports

• Add support for custom ports

• Lower required php version to 5.5

• Make url's case sensitive

• First release

• Ignore tel: links when crawling

• Added path, segment and segments functions to Url

• Updated the required version of Guzzle to a secure version

• Fixed a bug where the crawler would not take query strings into account

• Fixed a bug where the crawler tries to follow JavaScript links

• Keep only guzzlehttp/psr7 v2.0 by @flangofas in https://github.com/spatie/crawler/pull/392

• @flangofas made their first contribution in https://github.com/spatie/crawler/pull/392

Full Changelog: https://github.com/spatie/crawler/compare/7.0.4...7.0.5