Weave Code
Url Harvester Laravel Package
piedweb/url-harvester
URL Harvester is a PHP package for extracting and collecting URLs from HTML, text, or sitemaps. It scans content for links, normalizes and deduplicates results, and helps you quickly build link indexes, crawlers, or URL datasets.
[PHP] Harvest statistics and meta data from an URL or his source code (seo oriented).
Frequently asked questions about Url Harvester
- How do I install piedweb/url-harvester in a Laravel project?
- Install it with Composer using composer require piedweb/url-harvester. Because the package is published on Packagist under the PiedWeb namespace, Laravel package auto-discovery may register a service provider and facade automatically, but you should confirm this by reading the package composer.json and source. Documentation is thin, so budget a short spike to inspect the entry point and confirm the public API before wiring it into application code.
- Does piedweb/url-harvester support Laravel 10, 11, or 12?
- There is no published Laravel support matrix in the available material, so treat version compatibility as unverified. Check the composer.json require block for the PHP version ceiling and any illuminate or laravel constraints, then run composer update in a branch to see whether it resolves against your lockfile. The package is essentially a plain PHP utility, so it may work fine even without formal Laravel bindings.
- Can I use piedweb/url-harvester without Laravel?
- Yes. It is a Composer package that parses HTML, text, or sitemap input, so any PHP project can call it directly. The Laravel service provider and facade, if present, are convenience wrappers rather than a hard requirement. That also makes it easy to isolate behind your own interface if you later swap it out.
- Does piedweb/url-harvester render JavaScript before extracting links and meta data?
- No. It parses static HTML source, so anything injected client-side by JavaScript will not appear in the harvested links, title, meta description, or headings. For single-page apps or JS-heavy sites you need a headless browser such as Puppeteer, Playwright, or a rendering service, and then pass the rendered HTML into the harvester. Plan for that split early if your targets are modern front-end applications.
- What exactly does the harvested statistics and meta data return?
- Expect SEO-oriented signals such as the page title, meta description, headings, canonical URL, link counts, and word counts, based on the package description. The exact return shape is not documented or covered by a published schema, so read the source and write a small fixture-based test that pins the keys you depend on. Always normalize the raw array into your own DTO rather than passing it around your domain.
- Can I inject my own HTTP client, proxy, timeout, or user agent?
- This is unclear from the available documentation and should be confirmed by reading the source for Guzzle usage or a configurable client. If the package does its own fetching without a clean injection point, the safer pattern is to fetch the page yourself with a controlled Guzzle client and pass only the HTML source code into the harvester. That gives you timeouts, retries, redirect limits, and egress policy under your own control.
- Is piedweb/url-harvester safe and maintained enough for production use?
- It has very low adoption and effectively a single maintainer, so assume limited community testing and thin documentation. Pin the version, wrap it behind an internal interface returning your own normalized DTO, and add regression tests with saved HTML fixtures. Also guard the fetch layer against SSRF, redirect loops, and oversized responses, since outbound URL fetching is the riskiest part of this kind of package.
- How do I extract every URL from a sitemap or a block of HTML text?
- The package accepts a URL or raw source code and scans the content for links, then normalizes and deduplicates the results. Use it as the parsing step inside a queued job or artisan command, and store the output in your own link index table with an explicit source column. Validate and re-check URLs before following them, since harvested links can point anywhere.
- How should I test SEO extraction with this package in a Laravel feature test?
- Store a handful of representative HTML fixtures on disk and assert against your own SeoSnapshotService output rather than the package array directly. Fake outbound HTTP with Http::fake or a mocked Guzzle handler so tests never hit the network. Snapshot-style assertions on your DTO will catch silent return-shape changes when you upgrade the dependency.
- What are the alternatives to piedweb/url-harvester for SEO scraping in PHP?
- For a maintained baseline, combine Guzzle with Symfony DomCrawler and write the extraction selectors yourself, or use a crawler package such as spatie/crawler for link discovery. If your stack is polyglot, Python options like Scrapy or BeautifulSoup have far healthier ecosystems. Given the package is small, an honest outcome of a spike may be vendoring the few hundred lines you actually need instead of taking the dependency.
Weaver
How can I help you explore Laravel packages today?