Url Harvester Laravel Package

Getting Started

Install and confirm it loads. The package is small (a single-purpose library), so installation is the whole setup:

composer require piedweb/url-harvester

Check vendor/piedweb/url-harvester/composer.json for a extra.laravel.providers block. Packages this size often ship without a service provider or facade — meaning you instantiate classes directly rather than resolve()-ing them. If there is no provider, do not add the package to config/app.php; just use the class.

Read src/ before the README. With a package this small, the source is the documentation. Fifteen minutes in vendor/piedweb/url-harvester/src/ will tell you more than the README: what the entry class is called, what the return type is (array vs. DTO), and whether it fetches URLs itself or only parses HTML you hand it. That distinction drives everything else.

First use case — on-page SEO audit of a single page. The natural day-one job is: given a URL, get title, meta description, headings, word count, link/image counts. Wrap it in a tinker session before writing any app code:

use PiedWeb\UrlHarvester\Harvester; // verify exact namespace in src/

$data = (new Harvester('https://example.com/blog/post'))->getData();
dump($data);

If the class expects raw HTML instead of a URL, the pattern is:

$html = Http::get($url)->body();
$data = (new Harvester($html))->getData();

Confirm which mode you're in before building on top of it — the "URL or his source code" split in the description is the package's core design decision.

Implementation Patterns

Wrap it in your own service, not scattered calls. Given the maintenance risk (1 star, 0 dependents), treat the package as an implementation detail behind an interface:

interface PageAuditor {
    public function audit(string $url): PageAudit;
}

PageAudit is your own DTO. When the package stalls or you swap to symfony/dom-crawler + custom rules, only the adapter changes. This is the single highest-value pattern for a low-adoption dependency.

Two acquisition strategies — pick deliberately.

• Fetch mode: let the package (or Http::get()) retrieve the URL. Simpler, but you inherit its timeout/redirect/user-agent handling.
• Source-code mode: fetch HTML yourself with Laravel's HTTP client (Http::withHeaders(...)->timeout(10)->get()), then hand the body to the harvester. You get retries, ->retry(), caching, and proxy config from Laravel, and the harvester stays a pure parser. Prefer this for anything scheduled or queued.

Batch auditing via queue. Never loop over URLs in a request. One job per URL, throttled:

class HarvestUrl implements ShouldQueue
{
    public $tries = 3;
    public $backoff = [10, 60, 300];

    public function __construct(public string $url) {}

    public function handle(PageAuditor $auditor): void
    {
        $audit = $auditor->audit($this->url);

        PageAuditModel::updateOrCreate(
            ['url' => $this->url],
            ['data' => $audit->toArray(), 'harvested_at' => now()]
        );
    }
}

RateLimited middleware or Redis::throttle() around the job keeps you from getting IP-banned — and keeps the package's own HTTP layer (if used) out of the hot path.

Store time-series, not just latest state. SEO value comes from diffs: title changed, word count dropped, H1 disappeared. An audits table with url, data (JSON), harvested_at lets you show regressions. A updateOrCreate on url alone throws that history away.

Sitemap-driven crawl. Ingest sitemap.xml → dispatch jobs → audit. Cap concurrency and always check robots.txt first; crawling a client's staging site by accident is a real failure mode.

Admin surface. The output is tabular — a Filament/Nova resource listing URL, status, title, word count, missing-meta flag reads well and is a small amount of code.

Scheduled re-audit. A weekly php artisan schedule:run command that re-queues every tracked URL is the standard way this package earns its keep long-term.

Combine with a JS renderer for SPAs. The harvester sees only server HTML. For React/Vue sites, run the URL through a headless browser service and feed that HTML into source-code mode.

Gotchas and Tips

Maintenance risk is the headline. One star, zero dependents, and a tiny surface area mean: pin the version in composer.json ("piedweb/url-harvester": "^1.0" — check the actual tag), commit composer.lock, and add a test that asserts the harvester's output shape on a fixture HTML file. If upstream breaks or is abandoned, that test tells you immediately and the interface above makes replacement a day's work.

Encoding will bite you. French-authored SEO packages often assume UTF-8; real-world pages don't. If mb_* functions aren't used internally, you'll get mojibake in titles. Test with an ISO-8859-1 page. Check for <meta charset> handling in src/.

Redirects and status codes. Verify whether a 301 is followed and whether the final URL is reported. Canonical-URL auditing depends on it. If the package doesn't return the resolved URL, capture it yourself from the HTTP client response.

Timeouts are not yours by default. If the harvester uses file_get_contents() or raw cURL internally, you cannot configure timeouts or user agents. That's the strongest argument for source-code mode: fetch with Http::timeout(10)->retry(2, 200), parse separately.

Blocked responses look like empty data. A 403 or a Cloudflare challenge page parses into "no title, no description" — indistinguishable from a genuinely broken page. Always capture the HTTP status alongside the harvest and treat non-200 as "unknown", not "failing SEO". Log the raw body for a sample of failures.

JS-rendered content is invisible. Word counts on SPAs will read near-zero. Don't ship a dashboard that silently misreports client sites.

Don't hammer in dev. Iterating on a 200-URL batch repeatedly against a production site is a fast way to get blocked. Cache raw HTML to disk in development (Storage::put("fixtures/".md5($url).".html", $html)) and replay from fixtures — this also makes your tests fast and deterministic.

Extension points are limited — favor composition. With no facade, no config, and likely no events, the practical extension points are: (a) your adapter class, (b) a decorator that post-processes the returned array (e.g. strip trailing whitespace from titles, normalize relative URLs), (c) a subclass if the entry class is non-final. Prefer (a)/(b) — subclassing a 1-star package couples you to its internals.

Debugging checklist when output is empty: dump the raw HTML first, confirm strlen($html) > 0; confirm the HTML actually contains the tags you expect with grep; then step into src/ and dd() the extraction point. There is not enough community knowledge for Stack Overflow to help — the source is your only reliable reference.