terminal42/escargot
A Symfony HttpClient-based crawler framework for PHP that can crawl any HTTP resource, not just websites. Jobs are tracked by ID and backed by a queue, so long crawls can pause and resume.
Escargot 2.0 fixes transport exception handling and exposes redirect destinations to subscribers. A major release is required because of two compatibility changes:
ExceptionSubscriberInterface::onTransportException() now accepts ResponseInterface|null. Symfony can throw before returning a response, which previously caused a TypeError while handling the original exception. Existing subscriber implementations must update their signature and handle null.DoctrineQueue now requires a nullable redirected_to column. Existing queue tables must be migrated before using 2.0. Custom persistent queues must also store and restore redirect information. CrawlUri::getRedirectedTo() exposes the final redirect destination while getUri() retains the original requested URI. Redirect information is available to subscribers and persisted even when further response processing is skipped. The HTML crawler now resolves relative links against the final destination.Full Changelog: https://github.com/terminal42/escargot/compare/1.7.0...2.0.0
Full Changelog: https://github.com/terminal42/escargot/compare/1.6.3...1.7.0
Full Changelog: https://github.com/terminal42/escargot/compare/1.6.1...1.6.2
Escargot::withMaxDurationInSeconds() optionAlso see https://github.com/terminal42/escargot/pull/34
Full Changelog: https://github.com/terminal42/escargot/compare/1.5.3...1.6.0
Full Changelog: https://github.com/terminal42/escargot/compare/1.5.2...1.5.3
Full Changelog: https://github.com/terminal42/escargot/compare/1.5.1...1.5.2
Full Changelog: https://github.com/terminal42/escargot/compare/1.5.0...1.5.1
DoctrineQueue has a new getTableSchema() method to fetch the required schema separate from createSchema().robots.txt evaluation.Full Changelog: https://github.com/terminal42/escargot/compare/1.4.0...1.4.1
Full Changelog: https://github.com/terminal42/escargot/compare/1.3.0...1.4.0
data-escargot-ignore on links to exclude them completely from the HtmlCrawlerSubscriberEscargot has been around for almost 1.5 years and was installed over 110k times. It deserves a version 1.0.0 tag now! 🎉
How can I help you explore Laravel packages today?