Crawler Detect Bundle Laravel Package

Getting Started

This is a Symfony bundle. A Laravel developer almost never pulls it in directly — in a Laravel app the natural path is the engine it wraps, jaybizzle/crawler-detect (optionally with the Laravel bridge jaybizzle/laravel-crawler-detect). Use the bundle only if you're inside a Symfony app or maintaining a mixed/legacy codebase.

Where to look first

1. composer require nmure/crawler-detect-bundle — then check composer why jaybizzle/crawler-detect to confirm which version of the underlying engine got resolved.
2. In the repo, read the DI extension (DependencyInjection/) and Resources/config/services.* — the bundle's entire job is registering CrawlerDetect in the container. There is nothing else to learn here; the real API is jaybizzle/crawler-detect.
3. The library's Crawlers (regex list) and Exclusions (false-positive guard) classes are where behavior actually lives — read those when a specific user agent misclassifies.

First use case — the canonical one: a middleware that flags bot traffic so it doesn't pollute analytics, hit rate limits, or trigger cache-warming side effects.

// app/Http/Middleware/FlagCrawler.php
use Jaybizzle\CrawlerDetect\CrawlerDetect;
use Closure;

class FlagCrawler
{
    public function __construct(private CrawlerDetect $detector) {}

    public function handle($request, Closure $next)
    {
        $request->attributes->set('is_crawler', $this->detector->isCrawler($request->userAgent()));
        return $next($request);
    }
}

Bind it once in AppServiceProvider::register() with $this->app->singleton(CrawlerDetect::class) — the detector compiles a large regex and you do not want that per resolution.

Implementation Patterns

Register the engine as a singleton, not the bundle. In Laravel, mirror what the Symfony bundle does for you:

$this->app->singleton(CrawlerDetect::class, fn () => new CrawlerDetect());

Then inject it into middleware, jobs, and listeners. Avoid new CrawlerDetect() at call sites.

Memoize per request. isCrawler() runs a large alternation regex. If several middleware/listeners need the answer, compute it once and stash it on the request ($request->attributes) or in a request-scoped singleton. Repeatedly calling it in a loop over log lines is the classic performance trap.

Gate side effects, not just responses. The highest-value pattern is suppressing background work for crawlers: skip dispatch() of analytics events, skip queue jobs that would warm caches for a bot that won't execute JS, and skip writing "unique visit" counters.

if (! $request->attributes->get('is_crawler')) {
    VisitRecorded::dispatch($request);
}

Response shaping / prerender. Crawlers don't run JS, so route known bots to a server-rendered or prerendered view while real users get the SPA shell. Branch on is_crawler inside the controller or a response middleware.

Use getMatches() for observability, not just booleans. When a UA is flagged, log the matched pattern. That single field turns "why was my scraper blocked?" support tickets into a one-line answer.

Custom lists / extension points. The library exposes setUserAgent() and lets you subclass Crawlers to append organization-specific patterns (internal health-check bots, partner scrapers). Register your subclass in the container in place of the base class rather than editing vendor code. In Symfony you'd override the CrawlerDetect service definition; in Laravel, bind your subclass to the CrawlerDetect abstraction.

Testing pattern. Drive it through headers, not mocks:

$this->withHeader('User-Agent', 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)')
     ->get('/')
     ->assertSee('<meta name="prerendered"', false);

Keep a small fixture set of real UA strings (Googlebot, Bingbot, curl, a normal Chrome, and one spoofed Chrome with a Googlebot substring) so regressions in your middleware surface immediately.

Gotchas and Tips

• It's effectively a stale dependency. Last release December 2017 and it pins old Symfony constraints. On PHP 8 / modern Symfony, composer require will often fail or downgrade your stack. In Laravel, install the underlying library directly and skip the bundle entirely.
• The regex list is the product — keep it fresh. The bundle won't pull new crawler signatures on its own beyond its constraint. Pin jaybizzle/crawler-detect explicitly at ^1.2 and run composer update jaybizzle/crawler-detect periodically; new bots appear constantly.
• User agents lie. isCrawler() is a heuristic, not authentication. A malicious scraper sets a Chrome UA and sails through; a well-behaved bot may set nothing. Never make security, auth, or billing decisions on UA alone — combine with reverse-DNS verification (Googlebot, Bingbot publish their ranges) for anything that matters.
• Missing or empty UA. Requests with no User-Agent header are a real category (health checks, some proxies). Decide explicitly whether they count as crawlers in your code; don't inherit the library's default silently.
• False positives via the Exclusions list. Legitimate browsers occasionally match crawler patterns; the library maintains an exclusion set to compensate. If a real user gets blocked, inspect the matched pattern and check whether you're running an outdated exclusion list.
• Don't call it in hot loops. Per-log-line or per-record checks are a performance bug. Compute once per request/job and pass the boolean down.
• Version drift between bundle and library. Because the bundle only declares a bare dependency, two apps on "the same bundle" can resolve to very different crawler lists. Record the concrete jaybizzle/crawler-detect version in your lockfile and diff it across environments if bot behavior differs.
• Extension point is the container, not config. There is no meaningful bundle config knob for detection rules. To change behavior you replace the service with a subclass or decorate it — write that down for whoever inherits the code.
• Consider whether you need it at all. If your only goal is blocking obvious scrapers at the edge, Cloudflare or web-server-level UA rules may be simpler and cheaper than an in-app check that runs after PHP has already booted.