Arachnid Laravel Package

Getting Started

Arachnid is a thin, synchronous crawler: one Crawler object, one URL, one adapter that fetches HTML, and a parser that pulls links and SEO fields out of the response. Before writing any integration code, read three things in the repo (in this order):

1. src/Codeguy/Arachnid/Crawler.php — constructor signature, traverse(), and getUniqueLinks(). This is the whole public API.
2. src/Codeguy/Arachnid/Adapters/ — the adapter interface, the Guzzle adapter (default, static HTML), and the PhantomJS adapter (JS-rendered sites).
3. The SEO extraction layer — the fields it pulls (title, meta description, canonical, headings, etc.) and whether they are returned per page or as one collection.

Install and confirm dependency resolution early, because the package is old:

composer require codeguy/arachnid

Then the smallest useful call — crawl the homepage to depth 1 and dump the deduplicated internal links:

require 'vendor/autoload.php';

use Codeguy\Arachnid\Crawler;

$crawler = new Crawler('https://example.com');
$crawler->traverse();                  // check Crawler.php for depth/limit args

foreach ($crawler->getUniqueLinks() as $url) {
    echo $url, PHP_EOL;
}

First real use case: point it at a staging domain and eyeball getUniqueLinks() against your expected page inventory. That single run tells you whether the crawl scope, URL normalization, and depth handling match your expectations before you build a report on top of it.

Implementation Patterns

Swap the adapter, keep the crawler. The adapter is the seam that lets one code path handle static and JS-rendered sites. Keep the adapter selection in one place (a factory, or a config flag) so the rest of your code never branches on "is this an SPA":

use Codeguy\Arachnid\Crawler;
use Codeguy\Arachnid\Adapters\PhantomJS;

$crawler = new Crawler(
    $baseUrl,
    $site->is_spa ? new PhantomJS() : null   // null = default Guzzle adapter
);

Audit as an Artisan command. For day-to-day use, wrap the crawl in a command with a progress bar and a CSV/JSON writer. Persist one row per page with the extracted SEO fields, then diff runs to spot regressions (missing titles, duplicate meta descriptions, canonical drift):

$crawler = new Crawler($baseUrl);
$crawler->traverse();

$rows = [];
foreach ($crawler->getUniqueLinks() as $url) {
    $rows[] = ['url' => $url] + /* per-page SEO fields from Crawler */ [];
}

Storage::put('audits/'.now()->format('Y-m-d').'.csv', csv($rows));

One queued job per site, not one per page. The crawler is blocking and holds everything in memory, so isolate each audit in a job with $timeout and $tries set, and raise memory_limit if you crawl more than a few hundred pages. The scheduler (->weekly()) turns this into a recurring SEO regression check for free.

Fixture-based tests via a fake adapter. Implement the adapter interface to return canned HTML from tests/fixtures/*.html. You get deterministic tests for link extraction and SEO parsing without hitting the network, and it is the only practical way to unit-test crawler behaviour.

Link inventory / orphan-page detection. Export getUniqueLinks() and cross-reference against sitemap.xml. URLs in the sitemap and absent from the crawl are either orphaned or blocked; URLs in the crawl and absent from the sitemap are crawl-budget leaks. This is the highest-value, lowest-effort use of the package.

Normalize before you dedupe. Post-process links through a normalizer (lowercase host, strip fragments and tracking params, resolve relative URLs, unify trailing slash and http/https) and feed the normalized base URL into the constructor. Arachnid dedupes what it is given; it does not know your canonical policy.

Persistence shape. Store url, status, title, meta_description, canonical, h1_count, word_count, crawled_at. That schema supports duplicate-title reports, missing-meta reports, and thin-content reports without re-crawling.

Gotchas and Tips

• Dependency age is the first blocker. Last release 2021, and it predates modern Guzzle/Symfony DomCrawler majors. Expect constraint conflicts on PHP 8.2+/Laravel 10+. Check composer.json before committing; either pin an isolated service, fork and bump constraints, or use it only in a CLI tool separate from the app's dependency tree.
• PhantomJS is archived. The JS support depends on an external PhantomJS binary that has not been maintained since 2018. It works, but treat it as legacy: for modern SPAs prefer headless Chrome (Browsershot/Puppeteer) and reserve Arachnid for static HTML crawling.
• traverse() is blocking and unbounded by default. Always set a max depth and a page cap. Without one, pagination, calendars, and session-ID URLs produce effectively infinite crawls that exhaust memory before they finish.
• No robots.txt handling and no rate limiting. The crawler will happily hammer a site. Wrap the adapter to add usleep() between requests and honor robots.txt, and set a descriptive User-Agent — otherwise you will get 403'd halfway through a run.
• Relative links break internal-link detection. If the parser returns relative hrefs, the same-domain filter fails and the crawl stops early. Confirm links are resolved to absolute URLs; if not, resolve them yourself against the page URL and handle <base href>.
• URL equivalence is not handled for you. http vs https, www vs apex, trailing slash, and ?utm_source= are all distinct strings to the crawler. Pick a canonical base URL and normalize the collected links.
• Per-page failures can abort a run. Wrap the adapter/fetch call so a timeout or 500 on page 40 does not kill the whole audit; log the failure with the URL and continue.
• Auth, cookies, and custom headers are adapter concerns. There is no session support in the core. If you need to crawl behind a login, implement it in a custom adapter rather than subclassing the crawler.
• Empty SEO fields on JS sites usually mean "rendered too early." If title/meta come back blank via the JS adapter, the render did not wait for the app to mount — add a wait or switch to a headless-Chrome-based fetcher.
• Extension points are the adapter and the crawler subclass. The adapter is for changing how HTML is fetched; subclassing Crawler is for changing which links are followed or how dedupe works. There is no plugin system, no config file, and no event hooks — everything is constructor arguments and overrides, so keep your customizations in one small, well-documented subclass.
• Debug with the adapter, not the crawler. When results look wrong, dump the raw HTML your adapter returns for a single URL first. Nearly every "crawler bug" is either an empty JS-rendered body or a redirect the adapter followed silently.