j0k3r/graby
Graby extracts article content from web pages. It's a fully tested, Composer-friendly fork of Full-Text RSS built on php-readability and site_config, with decoupled HTTP clients. Requires PHP 8.2+ plus Tidy and cURL.
1. Install the package plus an HTTPlug client adapter — Graby ships no HTTP client of its own, so Composer will fail resolution if you only require j0k3r/graby:
composer require j0k3r/graby php-http/guzzle7-adapter
Pin the major you actually test against. The dev-master/3.x branch returns a Result object ($result->getHtml()), while the 2.x stable branch returns a plain array ($result['html']). Copy-pasted snippets from blogs are frequently 2.x and will silently break.
2. Verify the system dependencies. Graby needs the tidy and curl PHP extensions — tidy is the one that is usually missing:
php -m | grep -Ei 'tidy|curl'
# Debian/Ubuntu images (Sail, Forge, Docker):
# apt-get install -y php8.2-tidy php8.2-curl
Without tidy you get a hard failure inside the readability/extractor pipeline rather than a friendly message.
3. There is no Laravel service provider and no auto-discovery. You wire it yourself. Add a config file and a binding:
// config/graby.php
return [
'debug' => env('GRABY_DEBUG', false),
'log_level' => env('GRABY_LOG_LEVEL', 'info'),
'rewrite_relative_urls' => true,
'singlepage' => false, // each of these can cost extra HTTP requests
'multipage' => false,
'xss_filter' => false, // we sanitize with our own policy instead
'content_links' => 'preserve',
'error_message' => '[unable to retrieve full-text content]',
'allowed_urls' => [],
'blocked_urls' => [],
'http_client' => [
'ua_browser' => 'Mozilla/5.0 (compatible; MyApp/1.0)',
'default_referer' => 'https://myapp.test/',
'max_redirect' => 5,
],
'extractor' => [
'config_builder' => [
'site_config' => [storage_path('app/graby/site_config')],
],
'src_lazy_load_attributes' => ['data-src', 'data-lazy-src', 'data-original'],
],
];
// app/Providers/AppServiceProvider.php (bind, not singleton — see Gotchas)
use Graby\Graby;
use GuzzleHttp\Client as GuzzleClient;
use Http\Adapter\Guzzle7\Client as GuzzleAdapter;
$this->app->bind(Graby::class, function () {
$client = new GuzzleClient([
'timeout' => (int) env('GRABY_TIMEOUT', 10),
'connect_timeout' => 3,
'allow_redirects' => ['max' => config('graby.http_client.max_redirect')],
]);
return new Graby(config('graby'), new GuzzleAdapter($client));
});
4. First use case: a "paste a URL, get the article" endpoint. This is the single highest-value thing to ship first — it validates the whole pipeline in ten minutes.
Route::post('/extract', function (Graby $graby) {
$url = request()->validate(['url' => ['required', 'url']])['url'];
$result = $graby->fetchContent($url);
return response()->json([
'status' => $result->getResponse()->getStatus(),
'title' => $result->getTitle(),
'html' => $result->getHtml(),
'image' => $result->getImage(),
'authors' => $result->getAuthors(),
'language' => $result->getLanguage(),
'date' => $result->getDate(),
'summary' => $result->getSummary(),
]);
});
Where to look first in the source: README.md → the Full configuration block (it is the real API docs) → src/Graby.php → src/Extractor/ContentExtractor.php. Almost every "why did it strip my markup" question is answered by a config key in that block, not by code.
A cold fetch + Readability parse + Tidy pass routinely takes 1–10 seconds and does 1–3 outbound HTTP requests. Always dispatch it:
class ExtractArticle implements ShouldQueue
{
public int $tries = 3;
public int $timeout = 60; // must exceed your Guzzle timeout
public array $backoff = [10, 60, 300];
public function __construct(public string $url) {}
public function handle(Graby $graby): void
{
$result = $graby->fetchContent($this->url);
if ($result->getResponse()->getStatus() !== 200
|| $result->getHtml() === config('graby.error_message')) {
throw new RuntimeException("Extraction failed for {$this->url}");
}
Article::updateOrCreate(['url' => $this->url], [
'title' => $result->getTitle(),
'content' => $result->getHtml(),
'lead_image' => $result->getImage(),
'authors' => $result->getAuthors(), // json column
'language' => $result->getLanguage(),
'published_at' => $result->getDate(),
'is_native_ad' => $result->getIsNativeAd(),
'fetched_at' => now(),
]);
}
}
Cast the column ('authors' => 'array', 'published_at' => 'datetime') and you never touch Graby's raw output again.
Extracted content changes slowly. Wrap in Cache::remember keyed on a normalised URL, and flush the key when the user hits "refresh":
$payload = Cache::remember("graby:".sha1($url), now()->addHours(6), fn () =>
$this->toArray($graby->fetchContent($url))
);
For re-crawl jobs, key on sha1($url.$etag) or store fetched_at and skip if fresher than N hours. Never cache failed extractions for the same TTL — you'll freeze a transient 503 into a permanent "unable to retrieve" page.
Graby will fetch whatever you hand it, including http://169.254.169.254/latest/meta-data/. This is the one security control you must own:
private function assertFetchable(string $url): void
{
$host = parse_url($url, PHP_URL_HOST);
abort_unless(in_array(parse_url($url, PHP_URL_SCHEME), ['http', 'https'], true), 422);
abort_if(in_array($host, ['localhost', '127.0.0.1', '::1'], true), 422);
foreach (dns_get_record($host, DNS_A) as $record) {
abort_if(filter_var($record['ip'], FILTER_VALIDATE_IP,
FILTER_FLAG_NO_PRIV_RANGE | FILTER_FLAG_NO_RES_RANGE) === false, 422);
}
}
Then also set allowed_urls in config for high-trust deployments (internal tools, a fixed publisher list). blocked_urls is a blacklist and is ignored entirely whenever allowed_urls is non-empty — pick one model, don't layer both.
$graby->setContentAsPrefetched($htmlFromYourOwnClient);
$result = $graby->fetchContent($url);
cleanupHtml() for untrusted HTML you already haveWhen content arrives via email ingest, a CMS webhook, or an RSS body rather than a URL, skip the fetch and just sanitise:
$result = $graby->cleanupHtml($rawHtml, $canonicalUrl); // URL is still needed to absolutise links
$clean = $result->getHtml();
This is the right tool for "email newsletter → blog post" pipelines: it runs the same filters, absolutises relative URLs against $canonicalUrl, and applies content_links.
content_links => 'footnotes' turns inline anchors into numbered footnotes — excellent for feed/reader UIs and for legal defensibility when you're re-publishing third-party text.content_links => 'remove' for a plain-text-ish view; 'preserve' (default) if you're rendering inside your own DOM.rewrite_relative_urls => true is essential, otherwise images and links break the moment you render the HTML on your domain.src_lazy_load_attributes is what saves images from modern publishers; the defaults cover data-src/data-lazy-src/data-original, but add site-specific ones in your config as you find them.Drop a storage/app/graby/site_config/<hostname>.txt (FiveFilters site-config syntax) with body, title, strip, single_page_link, next_page_link directives. This is far cheaper than patching Readability filters, and it's how you fix the long tail of publishers that Readability mis-parses. Keep that directory in version control, not in storage — deploy it as a read-only asset path and point site_config at it.
# config/logging.php
'channels' => [
'graby' => [
'driver' => 'monolog',
'handler' => \Graby\Monolog\Handler\GrabyHandler::class,
'level' => env('GRABY_LOG_LEVEL', 'info'),
],
],
In Laravel you can then pull the records from the handler for a per-request debug panel ($handler->getRecords()), which beats trawling storage/logs/laravel.log. Keep debug => false and log_level => 'info' in production; debug dumps HTML at each pipeline step to log/graby.log relative to the current working directory.
RateLimiter::attempt("graby:{$host}", 1, fn () => dispatch(new ExtractArticle($url)), 2);
Publishers will 429 you quickly when a backfill queue is running, and a 429 surfaces as a valid-looking 200-with-error-content result (see Gotchas). Throttling is cheaper than retry logic.
fetchContent() does not throw on HTTP errors. A 404, a 429, or a 30-second timeout all come back as a "successful" call with getStatus() !== 200, getHtml() === '[unable to retrieve full-text content]', and getTitle() set to the error page's title. If your only check is try/catch, you will store error pages as articles. Always branch on status and on the error-message sentinel.
Don't register Graby as a singleton. setContentAsPrefetched() mutates instance state and is not cleared by fetchContent(). Under Octane, or in a queue worker processing many jobs, a singleton() binding means job #50 silently parses job #1's HTML. Use bind() (a fresh instance per resolution) or instantiate per job.
"No HTTP client found" is a missing adapter, not a config bug. php-http/guzzle7-adapter (or php-http/curl-client) must be installed explicitly; also check that the adapter's major matches your Guzzle. Graby 3 dropped Guzzle 5/6 — if your app is stuck on Guzzle 6 you need Graby 2 and the array-shaped API.
Timeouts need the injected client, not a config key. The http_client config block has max_redirect but no timeout. Without a configured Guzzle client and a timeout, a slow publisher can pin a queue worker indefinitely. Set timeout and connect_timeout on the client, and make the job's $timeout larger than the HTTP timeout so you get a clean exception rather than a killed worker.
xss_filter => true deletes iframes, embeds and inline styles. It's on by default. If you render the result in your own app, prefer xss_filter => false and sanitise with mews/purifier or an HTML Purifier policy you control — you'll get YouTube/Spotify embeds and figure captions that Graby's aggressive filter strips, while keeping the sanitisation auditable in one place.
date is a string, not a DateTime, and may be junk. Run date_parse($result->getDate()) (or Carbon::parse inside a try/catch) — Graby already validates, but formats vary wildly and timezone offsets are common. Cast to UTC on write; don't trust the publisher's timezone for ordering.
Tidy can restructure your markup. It repairs malformed publisher HTML, which is usually good, but it can also drop attributes or re-nest elements in ways that break a CSS selector you rely on downstream. When output looks "different from the source" and no filter explains it, test with Tidy disabled first.
Debug logging is a disk-filler and a PII leak. debug => true writes full HTML at each step to log/graby.log in the CWD, and log_level => 'debug' dumps HTML into your configured channel. Both are fine locally, both are incidents in production (copyrighted content + user URLs in logs).
Testing: Http::fake() will not intercept Graby. It uses HTTPlug, not the Laravel HTTP client, so Laravel's fakes are bypassed and your tests hit the network. Either inject a fake adapter ($this->app->bind(Graby::class, fn () => new Graby([], new FakeClient(...)))) or use setContentAsPrefetched() with committed HTML fixtures. The latter is simpler and covers the parsing logic, which is where your bugs will be.
Content-Type short-circuits surprise people. PDFs, ZIPs, images, audio and video are converted to a link (or excluded) per content_type_exc — you'll get a Result with a link and no body. If users report "empty extraction" on a URL, check the response headers before assuming Readability failed. Relatedly, header_only_clues extensions (.mp3, .zip, .jpg, …) trigger a HEAD request first, which some servers reject.
Memory on giant pages. Readability + Tidy on a 5 MB page can push a 128 MB PHP worker over the limit, and the failure looks like a random 500 rather than an extraction error. Cap acceptable response sizes at the client level or raise memory_limit for the extraction queue specifically — a dedicated graby queue with its own worker config is the cleanest answer.
singlepage/multipage multiply your request count and only trigger when single_page_link/next_page_link exist in a site config. They're off in the config I showed above; turn them on per-queue if you're ingesting multi-page features, and remember each one is another chance to be rate-limited or to walk into an infinite pagination loop (bounded by max_redirect, but still expensive).
Version pinning is not optional. The 3.x API is incompatible with 2.x (Result object vs array), the PHP floor is 8.2, and dev-master is a moving target. Pin ^3.0 (once released/tagged) and add one integration test that asserts $result->getHtml() exists — that single assertion documents the API shape for the next developer.
Legal posture. Graby makes it trivially easy to republish full third-party text. Store the canonical getResponse()->getEffectiveUri() (after redirects) alongside the content, honour robots.txt/ToS at the point you enqueue URLs, and use content_links => 'footnotes' so attribution survives the pipeline.
How can I help you explore Laravel packages today?