Getting Started

Minimal Setup

Install the package (2.x):

composer require terminal42/escargot:^2.0

Prepare the queue table (only if you use DoctrineQueue). 2.0 requires a nullable redirected_to column:
```
ALTER TABLE escargot ADD redirected_to VARCHAR(2048) DEFAULT NULL;
```
Run this before the first 2.0 crawl — the queue writes the column on every redirect it follows.

Define base URIs and queue:

use Terminal42\Escargot\Escargot;
use Terminal42\Escargot\BaseUriCollection;
use Terminal42\Escargot\Queue\InMemoryQueue;
use Nyholm\Psr7\Uri;

$baseUris = new BaseUriCollection();
$baseUris->add(new Uri('https://example.com'));
$queue = new InMemoryQueue();

$escargot = Escargot::create($baseUris, $queue);

Add subscribers (e.g., for HTML crawling):

$escargot->addSubscriber(new \Terminal42\Escargot\Subscriber\RobotsSubscriber());
$escargot->addSubscriber(new \Terminal42\Escargot\Subscriber\HtmlCrawlerSubscriber());
$escargot->addSubscriber(new class implements \Terminal42\Escargot\Subscriber\SubscriberInterface {
    public function shouldRequest($crawlUri, $currentDecision) {
        return self::DECISION_POSITIVE; // Always crawl
    }
    public function needsContent($crawlUri, $response, $chunk, $currentDecision) {
        return self::DECISION_POSITIVE; // Always load content
    }
    public function onLastChunk($crawlUri, $response, $chunk) {
        // Process response (e.g., save to DB)
    }
});

Start crawling:
```
$escargot->crawl();
```

Upgrading from 1.x — the two things that will break

onTransportException() signature changed to (CrawlUri $crawlUri, \Throwable $exception, ?ResponseInterface $response). Symfony can throw before any response exists, so the third argument is now nullable (it used to be a hard ResponseInterface, which caused a TypeError while handling the original exception). Update every implementation and null-check the response.
DoctrineQueue needs the redirected_to column (see step 2). Custom persistent queues must also persist and restore redirect information or redirect data will silently disappear between runs.

Everything else (subscriber decisions, tags, Escargot::createFromJobId()) is source compatible.

First Use Case: Simple HTML Crawler

// Initialize with a single URL
$escargot = Escargot::create(
    (new BaseUriCollection())->add(new Uri('https://example.com')),
    new InMemoryQueue()
);

// Add built-in subscribers for robots.txt and HTML parsing
$escargot->addSubscriber(new RobotsSubscriber());
$escargot->addSubscriber(new HtmlCrawlerSubscriber());

// Add a subscriber to process responses
$escargot->addSubscriber(new class implements SubscriberInterface {
    public function shouldRequest($crawlUri, $currentDecision) {
        return self::DECISION_POSITIVE; // Crawl everything
    }
    public function needsContent($crawlUri, $response, $chunk, $currentDecision) {
        return self::DECISION_POSITIVE; // Load full content
    }
    public function onLastChunk($crawlUri, $response, $chunk) {
        $requested = (string) $crawlUri->getUri();
        $final = $crawlUri->getRedirectedTo() ?? $requested; // 2.0
        echo "Crawled: {$requested} -> {$final}\n";
        // Extract data (e.g., with Symfony's DomCrawler)
    }
});

// Run the crawler
$escargot->crawl();

Implementation Patterns

1. Queue Management

For CLI/Testing: Use InMemoryQueue (ephemeral, fast).
```
$queue = new InMemoryQueue();
```
For Production: Use DoctrineQueue (persistent, database-backed). It now stores the redirect target alongside the URI.
```
$queue = new DoctrineQueue($pdoConnection); // table needs `redirected_to`
```

Hybrid Approach: Use LazyQueue to offload to DB only when needed.

$queue = new LazyQueue(new InMemoryQueue(), new DoctrineQueue($pdoConnection));
$escargot->crawl(); // Work in-memory first
$queue->commit($jobId); // Persist to DB later

2. Subscriber Workflows

Filtering Requests

Use shouldRequest() to control which URLs are crawled:

$escargot->addSubscriber(new class implements SubscriberInterface {
    public function shouldRequest($crawlUri, $currentDecision) {
        return $crawlUri->getUri()->getHost() === 'example.com'
            ? self::DECISION_POSITIVE
            : self::DECISION_NEGATIVE;
    }
    // ... other methods
});

Conditional Content Loading

Use needsContent() to avoid loading large responses:

public function needsContent($crawlUri, $response, $chunk, $currentDecision) {
    return $response->getStatusCode() === 200
        ? self::DECISION_POSITIVE
        : self::DECISION_NEGATIVE;
}

Tag-Based Routing

Add metadata to URIs via tags (e.g., skip nofollow links):

$escargot->addSubscriber(new class implements SubscriberInterface {
    public function shouldRequest($crawlUri, $currentDecision) {
        return !$crawlUri->hasTag('nofollow')
            ? self::DECISION_POSITIVE
            : self::DECISION_NEGATIVE;
    }
    // ...
});

Working with Redirects (new in 2.0)

getUri() stays the originally requested URI; getRedirectedTo() returns the final destination (or null if there was no redirect). Redirect information is persisted and made available to subscribers even if you declined further response processing in needsContent(), so it's safe to use for link auditing without downloading bodies.

public function shouldRequest($crawlUri, $currentDecision) {
    // Skip anything that redirects off-site before we even fetch the body
    $target = $crawlUri->getRedirectedTo();
    if ($target !== null && $target->getHost() !== 'example.com') {
        return self::DECISION_NEGATIVE;
    }
    return self::DECISION_POSITIVE;
}

public function onLastChunk($crawlUri, $response, $chunk) {
    if (null !== $redirect = $crawlUri->getRedirectedTo()) {
        Log::info('Redirect', [
            'from' => (string) $crawlUri->getUri(),
            'to'   => (string) $redirect,
        ]);
    }
}

Typical 2.0 use cases this unlocks:

Storing the canonical URL in your index instead of the requested one.
Detecting redirect chains / loops and domain migrations between crawls.
Attributing content to the final URL so the same page isn't indexed twice (getUri() differs, getRedirectedTo() matches).
Auditing redirects cheaply — you get the destination even when needsContent() returned DECISION_NEGATIVE.

Note the built-in HtmlCrawlerSubscriber now resolves relative links against the final destination, so <a href="/foo"> on a page reached via http://example.com → https://www.example.com/ is crawled as https://www.example.com/foo. If you wrote your own link extraction against getUri(), mirror this behaviour to stay consistent.

3. Integration with Laravel

Service Provider

public function register()
{
    $this->app->singleton(Escargot::class, function ($app) {
        $queue = new DoctrineQueue($app['db']->connection()->getPdo());
        return Escargot::create(
            (new BaseUriCollection())->add(new Uri(config('escargot.start_url'))),
            $queue
        );
    });
}

Migration

Ship the new column with a standard Laravel migration so deploys stay in sync with the package:

Schema::table('escargot', function (Blueprint $table) {
    $table->string('redirected_to', 2048)->nullable()->after('uri');
});

Deploy order matters: migrate, then run crawlers. Running a 2.0 crawl against a 1.x table will fail on insert.

Artisan Command

Create a crawler command:

php artisan make:command CrawlWebsites

// In CrawlWebsitesCommand.php
public function handle()
{
    $escargot = app(Escargot::class);
    $escargot->addSubscriber(new MySubscriber());
    $escargot->crawl();
}

Job Resumption

Resume a paused crawl:

$escargot = Escargot::createFromJobId($jobId, $queue);
$escargot->crawl(); // Continues from where it left off

4. Handling Exceptions

Transport Errors (Timeouts, DNS) — signature changed in 2.0

use Psr\Http\Message\ResponseInterface;

$escargot->addSubscriber(new class implements ExceptionSubscriberInterface {
    public function onTransportException($crawlUri, $exception, ?ResponseInterface $response) {
        // $response is NULL when the failure happened before any response existed
        Log::error("Failed to fetch {$crawlUri->getUri()}: " . $exception->getMessage(), [
            'status' => $response?->getStatusCode(),
            'redirected_to' => (string) ($crawlUri->getRedirectedTo() ?? ''),
        ]);
    }
});

HTTP Errors (4xx/5xx)

$escargot->addSubscriber(new class implements ExceptionSubscriberInterface {
    public function onHttpException($crawlUri, $exception, $response, $chunk) {
        if ($response->getStatusCode() === 404) {
            Log::debug("Page not found: {$crawlUri->getUri()}");
        }
    }
});

5. Lazy-Loaded Tags

Store dynamic data (e.g., API responses) without bloating the queue:

// Subscriber 1: Adds a tag
$escargot->addSubscriber(new class implements SubscriberInterface {
    public function onLastChunk($crawlUri, $response, $chunk) {
        $crawlUri->addTag('api-data', 'lazy');
    }
});

// Subscriber 2: Resolves the tag
$escargot->addSubscriber(new class implements TagValueResolvingSubscriberInterface {
    public function resolveTagValue($tag, $crawlUri) {
        if ($tag === 'api-data') {
            return $this->fetchExternalData($crawlUri->getUri());
        }
    }
});

Gotchas and Tips

1. Breaking Changes to Check Before Deploying 2.0

Grep for onTransportException across app/ — every implementation must accept ?ResponseInterface as the third parameter, or PHP throws a signature/TypeError at the worst possible moment (while handling another exception). This includes anonymous classes registered in service providers.
Never call $response->getStatusCode() unguarded inside onTransportException(); DNS failures and connection resets arrive with null.
Run the redirected_to migration first. Existing escargot tables without the column will break DoctrineQueue writes.
Custom queue implementations (anything not InMemoryQueue / DoctrineQueue / LazyQueue) must now persist and restore redirect information. If you built a Redis queue, add the field to your payload; otherwise getRedirectedTo() returns null after a resume and your canonicalisation silently regresses.

2. Queue Persistence Pitfalls

InMemoryQueue: Data is lost on process exit. Use only for testing.
DoctrineQueue: Requires a PDO connection and the migrated schema. Configure in Laravel via:
```
$queue = new DoctrineQueue($app['db']->connection()->getPdo());
```
LazyQueue: Call $queue->commit($jobId) explicitly to persist data.

3. Performance Tips

Concurrency: Use Symfony's HttpClient with concurrency:

$client = new CurlHttpClient(['max_concurrency' => 10]);
$escargot = Escargot::create($baseUris, $queue, $client);

Rate Limiting: Implement a subscriber to throttle requests:

$escargot->addSubscriber(new class implements SubscriberInterface {
    private $lastRequestTime = 0;
    public function shouldRequest($crawlUri, $currentDecision) {
        $now = time();
        if ($now - $this->lastRequestTime < 1) { // 1 request per second
            return self::DECISION_NEGATIVE;
        }
        $this->lastRequestTime = $now;
        return self::DECISION_POSITIVE;
    }
    // ...
});

Skip bodies, keep redirects: needsContent() returning DECISION_NEGATIVE still gives you status codes and redirect targets — a cheap way to audit a large site's URL structure.

4. Debugging

Log Subscriber Decisions: Add a debug subscriber:

$escargot->addSubscriber(new class implements SubscriberInterface {
    public function shouldRequest($crawlUri, $currentDecision) {
        Log::debug("Should request {$crawlUri->getUri()}? Decision: {$currentDecision}");
        return self::DECISION_ABSTAIN; // let other subscribers decide
    }
    public function needsContent($crawlUri, $response, $chunk, $currentDecision) {
        return self::DECISION_ABSTAIN;
    }
    public function onLastChunk($crawlUri, $response, $chunk) {}
});

Log requested vs. final URI side by side. A bare getUri() in logs makes redirect-heavy sites look like the crawler is fetching pages it never actually lands on.
Empty redirected_to after resume? Suspect an unmigrated table or a custom queue that drops the field — not a bug in Escargot.
Unexpected off-site crawls after 2.0? Relative links now resolve against the final destination, so a redirect to another host widens the effective base. Tighten shouldRequest() with an explicit host allow-list if that's not what you want.

Escargot Laravel Package