Arachnid Laravel Package

Product Decisions This Supports

• Build vs. buy for SEO auditing. Informs whether to stand up an in-house crawl/audit capability on top of a PHP library versus paying per-seat/per-domain for Screaming Frog, Sitebulb, Lumar (DeepCrawl), or Ahrefs Site Audit. Arachnid covers the "crawl internal links, extract title/meta/canonical/H1" core that covers the majority of recurring SEO QA needs.
• Roadmap scoping for an SEO or content-ops feature. If the product is a CMS, marketing site builder, or agency tooling platform, this package is a credible foundation for a "site health" or "SEO preview" feature without licensing a third-party API — useful for deciding whether SEO auditing is a v1 or v2 feature.
• Content inventory and migration QA. Concrete use case for site relaunches: enumerate all reachable internal URLs, capture titles/metas/canonicals before a migration, diff against the post-launch crawl to catch dropped redirects, orphaned pages, and duplicate metadata. This is a recurring, high-cost failure mode that justifies tooling spend.
• Competitive and content gap research. Crawling a competitor's public site to inventory page titles, headings, and information architecture supports positioning, content-gap analysis, and IA benchmarking — decisions that otherwise get made on gut feel.
• Internal-tooling prioritization. Because it is MIT-licensed and self-hosted, it enables decisions about where crawl data may live (no third-party data egress), which matters for regulated industries and for products where customer URLs are sensitive.
• Feature parity triage. Gives engineering a realistic cost estimate for "SEO checks" — the package proves the extraction layer is cheap, so roadmap arguments should center on the expensive parts (JS rendering at scale, scheduling, diffing, reporting UX), not on crawling itself.

When to Consider This Package

Adopt when:

• The stack is PHP/Laravel and the team wants crawl data inside an existing application, job queue, and database rather than a separate SaaS tool.
• The need is internal-link crawling plus on-page metadata extraction — titles, meta descriptions, canonicals, headings — not full technical SEO suites (log-file analysis, Core Web Vitals, backlink indices, rank tracking).
• Data residency, per-domain costs, or volume make SaaS crawling uneconomic — e.g., auditing thousands of customer sites or running nightly crawls across a portfolio.
• The team is comfortable owning a thin integration layer and writing the persistence, scheduling, diffing, and reporting around a library rather than expecting a turnkey product.
• Requirements are modest in scale (hundreds to low thousands of pages per site) and can run as background jobs.

Look elsewhere when:

• Maintenance risk is unacceptable. Last release was October 2021; there is no evidence of ongoing PHP 8.x modernization, dependency bumps (Guzzle/Symfony DomCrawler), or security patching. For anything customer-facing or long-lived, treat it as a fork-and-own commitment, not a dependency.
• JavaScript-heavy sites are the primary target. The "supports JavaScript-based sites" claim depends on an external headless renderer, and that ecosystem (notably PhantomJS) has been dead for years. Modern SPA/CSR sites need a maintained Puppeteer/Playwright/Chrome DevTools pipeline; budget for replacing the rendering layer entirely.
• You need scale or concurrency guarantees. There is no built-in distributed crawling, politeness/rate-limit orchestration at scale, robots.txt/sitemap-driven discovery, incremental recrawling, or render-budget management. Above roughly tens of thousands of pages these become the dominant engineering cost.
• You need a Laravel-native package. This is a framework-agnostic PHP library — no service provider, config publishing, facade, or queue integration. Laravel teams should price the wrapper work explicitly.
• The requirement is a real SEO product. Rank tracking, backlink data, Search Console/GA integration, historical trending, and Lighthouse metrics are out of scope and not cheaply bolted on.
• The team is not PHP. The crawl loop is trivial to reimplement in Node or Python with actively maintained tooling; porting is often cheaper than carrying an unmaintained PHP dependency.

How to Pitch It (Stakeholders)

To executives: Arachnid is a free, permissively licensed PHP crawler that walks a site's internal links and pulls the SEO metadata off each page. It is the cheapest credible way to answer "do we build our own SEO audit capability or keep renting one?" — a proof-of-concept is days of work, not a quarter, and it keeps customer URLs and crawl data in-house with no per-domain licensing fees. The catch is that it is a library, not a product: scheduling, storage, diffing, and reporting are ours to build, and the project has not shipped a release since 2021, so we would be adopting it as a codebase to own rather than a dependency to track. Recommendation: use it to validate demand and cost, and decide on the surrounding engineering investment — not the crawler — as the real build-vs-buy question.

To engineering: Treat this as an extraction primitive, not a crawling platform. It gives us a Guzzle-based traversal of unique internal links plus DOM-based metadata extraction; everything that makes a crawl operationally real — robots.txt handling, rate limiting, retries, queue orchestration, incremental recrawls, JS rendering, storage schema, and diffing — is on us. Two hard constraints to plan around: the JavaScript support is tied to a rendering integration whose underlying tooling is obsolete and should be replaced with a headless Chrome service behind an interface, and the 2021 release cadence means we should expect to vendor or fork it, pin dependencies deliberately, and budget for PHP 8.x compatibility work. The upside is that the surface is small, the code is readable, and replacing it later behind a Crawler interface is a contained refactor if we design the boundary correctly from day one.

Summary position: Worth a time-boxed spike as an internal SEO/migration-QA and site-health capability, contingent on a clean abstraction boundary and an explicit decision to own maintenance. Not appropriate as a customer-facing crawling engine or as the basis of a commercial SEO product without substantial additional investment.