Getting Started

Install and publish config

composer require illuminated/wikipedia-grabber
php artisan vendor:publish --tag=wikipedia-grabber.config

The config file (typically config/wikipedia-grabber.php) is small — the important key is the locale, which decides which Wikipedia edition you hit (en, de, ru, …). Set it from an env var so staging and production don't drift:

'locale' => env('WIKIPEDIA_LOCALE', 'en'),

Where to look first

The README on GitHub — it's the only real documentation.
Illuminate\Wikipedia\Grabber\Page — this is the object you'll spend 90% of your time with. Let IDE autocompletion on it be your API reference; the package is small enough that the class itself is the docblock.
Illuminate\Wikipedia\Grabber\Templates\* — the Plain and Common templates control how parsed sections are rendered. Read these before you write any content-massaging code, because they probably already do most of it.

First use case: article to text in one call

The canonical entry point is grabbing a page and dumping its rendered contents:

use Illuminated\Wikipedia\Wikipedia;

$page = Wikipedia::page('Albert Einstein');

if ($page->isExists()) {
    echo $page->getTitle();
    echo $page->getContents();   // rendered via the default template
}

Run that in php artisan tinker before you design anything. It tells you immediately what the output shape looks like — HTML-ish text, section headings inline, references stripped or not depending on template — and that shape drives every downstream decision.

A good first real task (not a toy) is an artisan command that grabs a handful of titles and writes them to storage/app/wiki/*.txt. That gets you a fixture corpus you can iterate against without hammering the API, which is exactly what you want before building anything bigger.

Implementation Patterns

Wrap it — don't call the facade from controllers

final class WikipediaArticleFetcher
{
    public function fetch(string $title): ?ArticleData
    {
        $page = Wikipedia::page($title);

        if (! $page->isExists() || $page->isDisambiguation()) {
            return null;
        }

        return new ArticleData(
            title: $page->getTitle(),
            summary: $this->summary($page),
            body: $page->getContents(),
            images: collect($page->getImages())->map->getUrl()->all(),
        );
    }
}

The package has 47 stars and 0 dependents. That's not a reason to avoid it, but it is a reason to have a single seam where you touch it, so a future swap is a one-file change.

Summary extraction

Most "day-to-day" use is not the whole article — it's the lead paragraph. Grab the sections, take the first one (usually untitled / level 1 before the first <h2>), and use it as a deck:

$sections = $page->getSections();
$lead = $sections[0] ?? null;
$summary = $lead ? strip_tags($lead->getContents()) : null;

Cache that aggressively — the lead paragraph of a famous person changes maybe twice a decade.

Bulk import as queued jobs

Never loop over 5,000 titles in a request. One job per title, with a throttle:

class ImportWikipediaPage implements ShouldQueue
{
    public $tries = 3;
    public $backoff = [30, 120, 600];

    public function __construct(public string $title) {}

    public function handle(ArticleRepository $repo): void
    {
        $data = app(ArticleFetcher::class)->fetch($this->title);
        if ($data) {
            $repo->upsert($data);
        }
    }
}

Then Bus::batch($titles->map(fn ($t) => new ImportWikipediaPage($t))) with ->allowFailures(). MediaWiki will return errors for some titles (deleted, moved, oddly encoded) and you don't want one bad title to kill a 2-hour import.

Pair that with RateLimiter::for('wikipedia', fn () => Limit::perMinute(60)) — or better, ->throttle() on the batch. Wikipedia tolerates scripted access but notices hammering.

Cache in front, always

$html = Cache::remember("wiki:{$locale}:{$title}", now()->addWeek(), function () use ($title) {
    return Wikipedia::page($title)->getContents();
});

The data is extremely cache-friendly — it's an encyclopedia. Treat any uncached fetch as a code smell.

Multi-locale content

The locale is global config, which means you can't grab en and de of the same article in one process without flipping config between calls. If you need cross-language, either:

config(['wikipedia-grabber.locale' => 'de']);
$de = Wikipedia::page('Albert Einstein')->getContents();
config(['wikipedia-grabber.locale' => 'en']);
$en = Wikipedia::page('Albert Einstein')->getContents();

…or run each locale in a separate job/process. Be aware that the title itself is localized (Albert Einstein vs Альберт Эйнштейн), so you need per-locale title lists — you can't just change the locale and reuse an English slug.

Custom templates for output shaping

Rather than post-processing HTML with regexes, subclass/extend the shipped templates. This is the package's real extension point — the grabber parses, the template renders. If you need Markdown, plain text for embeddings, or a stripped-down JSON structure, a custom template is far more maintainable than string surgery on the default output. Register it on the page and reuse it everywhere.

RAG / embedding pipeline

foreach ($page->getSections() as $section) {
    if (strlen($text = strip_tags($section->getContents())) < 200) {
        continue; // skip "See also", "References", navboxes
    }
    Embedding::create(['title' => "{$page->getTitle()} § {$section->getTitle()}", 'text' => $text]);
}

Section-level chunks beat one giant document — Wikipedia's H2 structure is already good chunking. Filter out the boilerplate sections or your vector store fills with citation lists.

Gotchas and Tips

Http::fake() will not intercept this. The illuminated packages talk to Guzzle directly rather than going through Laravel's HTTP client, so Laravel's fake won't catch the request — your tests will go out to the live network (or fail in CI). Stub at the Guzzle handler level, or better, put an interface in front of your fetcher and fake that in tests, keeping the real HTTP call in one integration test marked @group external.

Send a real User-Agent. Wikipedia's API etiquette expects it, and requests without one are more likely to be throttled or blocked. If the package doesn't expose it in config, that's a one-line change in your own client wrapper — worth doing before you run any batch.

Disambiguation and redirects are the two failure modes. isExists() isn't enough — a disambiguation page exists, has content, and is useless. Redirects (e.g. Einstein → Albert Einstein) behave differently depending on whether you're using the API's redirect resolution or not. Always check isExists() and isDisambiguation(), and for a title you care about, assert that getTitle() returned what you expect.

Section nesting is not flat. Wikipedia has H2/H3/H4. The section objects carry their level, and naively iterating gives you a list where subsections appear as siblings. If you build a table of contents or chunk for embeddings, walk by level and attach children to their parent, or you'll get a "History" chunk that starts mid-topic.

Content is HTML, and it's messy. <sup> citation markers, <span class="mw-editsection">, <math>, nested tables in infoboxes. strip_tags() leaves entity-encoded junk and glued-together words. If you're going to plain text, run it through html_entity_decode and insert whitespace at block boundaries before stripping, otherwise "Einsteinwasborn" happens.

Locale quirks. Changing the locale changes the API host, the namespace names, and the section names — so any logic keyed on "References" or "See also" breaks in German. Match on section position or level instead of literal titles.

Large pages can be genuinely large. Some articles return megabytes of HTML. Set a timeout in config and consider bumping memory_limit if your job renders many pages before flushing. If you only need the intro, don't call getContents() on the whole page — grab sections and take what you need.

No built-in caching (verify before assuming). If a config key for cache exists in your version, use it; otherwise wrap every call. It is very easy to accidentally write a loop that makes 400 requests to Wikipedia in a test suite.

Pin the version and watch the family. This package ships from a family of "grabber" libraries (laravel-grabber, google-news-grabber, etc.) that share internals. A minor bump in a sibling package can change behavior here. composer.json with a ^ constraint on the grabber but a review of the changelog on update is the sane middle ground.

Debugging tip: dump the raw parse before the template. When output looks wrong, the question is always "did the parser mis-read, or did the template mis-render?" Render with the plainest template available to separate the two. Nine times out of ten it's the template, and the fix belongs in your custom template rather than in regexes in your service class.

Test against saved fixtures. Grab ten representative pages once (a normal article, a disambiguation page, a redirect, a page with a big infobox, a missing title), store the raw responses as fixtures, and unit-test your parsing/chunking against those. You do not want network flakiness in your CI, and Wikipedia's markup does change over time — a fixture set tells you when it did.

Wikipedia Grabber Laravel Package

Getting Started

Implementation Patterns

Gotchas and Tips