Symfony Bundle Laravel Package

Getting Started

This release adds two first-class "reader" drivers — Jina Reader (hosted r.jina.ai URL-to-markdown service) and vLLM (any OpenAI-compatible self-hosted endpoint) — alongside HTML-based custom extractors for sites you want to parse deterministically instead of via an LLM.

Start here, in this order:

1. Publish the config — php artisan vendor:publish --tag="<vendor>-config" and read the readers / drivers block first. That block is the whole mental model of the package: an ordered map of driver name → options, plus a default key.
2. Pick the driver for your environment. jina needs JINA_API_KEY and gives you zero-infra markdown extraction; vllm needs VLLM_BASE_URL + VLLM_MODEL pointing at your own server and gives you no per-request cost and no data leaving your network.
3. Run one extraction end to end before touching custom extractors:

$doc = Reader::make()          // uses config('...default')
    ->read('https://example.com/some-article');

$doc->markdown();   // normalized markdown body
$doc->title();
$doc->metadata();   // driver-dependent, see gotchas

4. Then add one custom extractor. This is the real day-to-day use case: you have a handful of high-traffic domains you scrape repeatedly, and paying LLM latency for a static blog layout is wasteful. Register a selector-based extractor for those and let vLLM/Jina handle the long tail.

The package version is 0.1, so treat the public API surface as narrow-but-settling: read(), the Document value object, and the extractor registry are the parts you should depend on. Anything else, wrap it.

Implementation Patterns

Driver-per-call, not driver-globally. The default driver should be whichever one you can afford to fail. For most apps that's jina for exploratory work and vllm in production, or vice versa if you can't ship content to a third party. Select explicitly at the call site rather than flipping config between jobs:

Reader::driver('vllm')->read($url);
Reader::driver('jina')->read($url);

Fallback chain. Neither driver is reliable in isolation — Jina rate-limits and can return a degraded page for JS-heavy sites; a self-hosted vLLM instance is one OOM away from a 500. Wrap in a small service of your own that tries the custom extractor first, then the LLM driver, then the hosted reader:

class ArticleReader
{
    public function read(string $url): ?Document
    {
        foreach (['extractor', 'vllm', 'jina'] as $driver) {
            try {
                return Reader::driver($driver)->read($url);
            } catch (ReaderException $e) {
                report($e);
            }
        }
        return null;
    }
}

Catch the package's exception type, not \Throwable — you want genuine bugs to surface.

Custom extractors as a whitelist, keyed narrowly. Register on hostname plus path prefix, not bare hostname. A blog.example.com extractor that matches on host alone will silently mangle /docs and /pricing the moment the site restructures. Keep the selector logic boring — one content root, one title, one date — and return null when the root selector misses so your fallback chain kicks in rather than storing an empty document.

Queue everything. Both drivers are network-bound with multi-second tail latency. A job with $timeout = 120, $tries = 3, and backoff() of [10, 60, 300] is the shape you want. Set retryUntil rather than relying on tries alone if the upstream can be down for minutes.

Cache by content identity, not URL. Store $doc->markdown() under a hash of the URL and driver name, with a TTL of hours-to-days. Re-running the same URL through vLLM on every page view is the single easiest way to turn a cheap feature into an expensive one.

Sanitize before you render. Markdown coming back from either driver is attacker-influenced input if you're reading arbitrary URLs. Run it through your existing HTML purifier after converting back to HTML, or render with a CommonMark config that disables raw HTML. Do not blade-echo it.

Fake it in tests. Assert on driver selection and on the shape of the Document, not on extracted prose — the prose will change the first time you upgrade a model or the vendor tweaks their extractor. A container binding that returns a fixture Document keeps your suite fast and offline.

Gotchas and Tips

• vllm is not an endpoint, it's a contract. Anything OpenAI-compatible works in principle; in practice the package sends a specific prompt and parses a specific response shape. If your server ignores response_format or wraps output in a chatty preamble, extraction degrades quietly into "the model talked about the page" rather than failing. Smoke-test with a page you know cold before trusting it.
• Context window truncation is silent. Long pages exceed the model's window and you get a plausible-looking but partial result. Check whether the driver surfaces a truncated flag on the document; if it doesn't, compare input length against your model's limit yourself and route long pages to the HTML extractor path.
• Jina Reader is a network dependency with a shared rate limit. Your throughput is not yours to control. It also has a caching layer of its own, so a "stale" result may be Jina's cache, not yours — pass their cache-bypass option (usually a header via the package's header config) when freshness matters, and expect the cost.
• Custom extractor selectors rot. Any DOM-driven extractor is a maintenance liability with a expiry date you don't control. Add a cheap assertion — if the extracted body is under N characters, treat it as a miss and fall through. Log the misses; that log is your early warning that the site shipped a redesign.
• Metadata keys are driver-dependent. published_at, author, language may or may not be populated. Never ->metadata('author') into a non-nullable column. Treat metadata as best-effort enrichment and default it at the boundary of your own code.
• Config caching will bite you. php artisan config:cache in deploy means env() inside the package config returns null. Confirm the package's config file uses env() only at the top level and that your deploy pipeline re-caches; then debug "driver not found" by checking config('...readers') at runtime, not your .env.
• Extension points worth knowing: the extractor registry (register your own via the service provider, not by editing vendor config), the HTTP client binding (swap in one with your own retry/timeout/user-agent so requests are attributable and debuggable), and the driver resolution itself — binding a custom driver is usually cleaner than forking the package when you need a third backend.
• Pin the version. 0.1 with a feature list this broad means the driver option names will move. Constrain to 0.1.* and read the diff before every bump; the failure mode here is a silently-changed option key that falls back to a default and doubles your inference bill.