Crawler Detect Laravel Package

Getting Started

CrawlerDetect is a single-class, zero-dependency PHP library. Composer install is the whole setup — there is no service provider, no config file, no migrations.

composer require jaybizzle/crawler-detect

Look here first, in this order:

1. src/CrawlerDetect.php — the whole public API is four methods: isCrawler(), getMatches(), getUserAgent(), getHttpFrom().
2. src/Fixtures/Crawlers.php — the $data array of ~thousands of regex fragments. When a bot slips through, this file is the source of truth for "is it already known?"
3. src/Fixtures/Exclusions.php — the deny-list of exclusions used to suppress false positives (headless-ish browser UAs, YaBrowser-style strings). If a real user is being flagged, this is the file that decides why.
4. raw/Crawlers.txt — a flat, greppable export. Fastest way to answer "does this UA exist in the list?" without reading regex.

First use case — 60 seconds in tinker:

use Jaybizzle\CrawlerDetect\CrawlerDetect;

$d = new CrawlerDetect;
$d->isCrawler('Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'); // true
$d->getMatches(); // the fragment that matched — debugging only

Then the thing you almost certainly actually want: flag the request once, in middleware, and read the flag everywhere else.

// app/Http/Middleware/DetectCrawler.php
public function handle(Request $request, Closure $next): Response
{
    $detect = new CrawlerDetect($request->headers->all());

    $request->attributes->set('is_crawler', $detect->isCrawler());
    $request->attributes->set('crawler_match', $detect->getMatches());

    return $next($request);
}

Register it in bootstrap/app.php (->withMiddleware(fn ($m) => $m->append(DetectCrawler::class))) and you have a request-scoped verdict with no container gymnastics.

If you want a facade and a config stub, the community wrapper jaybizzle/laravel-crawler-detect wraps this package — but for most apps the four lines above are less machinery than the wrapper, and you skip a dependency that lags the base package's signature updates.

Implementation Patterns

Middleware flag, not middleware block. Set an attribute rather than aborting. Blocking decisions belong in a policy, because "is a crawler" and "is unwanted" are different questions (see Gotchas).

// Downstream: controllers, listeners, view composers
if ($request->attributes->get('is_crawler')) { /* ... */ }

Keep ads and heavy JS off crawler pages. The classic use case: crawlers should get clean, fast, semantic HTML.

@unless (request()->attributes->get('is_crawler'))
    <script src="{{ mix('js/analytics.js') }}"></script>
    <div id="ad-slot"></div>
@endunless

Better: View::share('isCrawler', ...) in a view composer so Blade stays readable.

Pass the full header set, not just the UA string. Since v1.4 the constructor accepts a headers array and understands both User-Agent and HTTP_USER_AGENT keys, with string-or-array values. This is not a nicety — Googlebot sends a genuine Chrome UA and identifies itself via From / Sec-CH-UA. A single-string check silently misses it.

new CrawlerDetect($request->headers->all());              // Symfony HttpFoundation
new CrawlerDetect($request->getHeaders());                // PSR-7
new CrawlerDetect($request->header);                      // Swoole
new CrawlerDetect($event['headers']);                     // Lambda / API Gateway

Prefer this over isCrawler($request->header('User-Agent')) in every case where you have a request object.

Split rate limits and cache keys by crawler status. Cheap, high-value, and it stops aggressive SEO crawlers from eating your per-IP budget shared with real users.

RateLimiter::for('web', fn (Request $r) => $r->attributes->get('is_crawler')
    ? Limit::perMinute(10)->by($r->ip())
    : Limit::perMinute(60)->by($r->user()?->id ?: $r->ip()));

Filter analytics and session recording. Drop bot traffic before it hits your page_views table or your session-replay budget. One is_crawler column on the request log is enough to filter retroactively.

Log the match string for observability, once. Write crawler_match into your slow/blocked request log for a week. If you see the same unrecognised UA repeatedly, you have a concrete case for a PR to src/Fixtures/Crawlers.php (add the regex + a line in tests/data/user_agent/crawlers.txt) rather than a local hack. raw/Crawlers.json and raw/Crawlers.txt are regenerated by export.php — never edit them by hand.

Test it with withHeader. Laravel's test request populates the server bag, so $_SERVER-based detection works in feature tests with no extra wiring.

$this->withHeader('User-Agent', 'Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)')
     ->get('/')
     ->assertDontSee('ad-slot');

App-specific signatures: wrap, don't fork. There is no publishable config and no addCrawler() documented surface you should lean on. Put your handful of internal UAs (uptime monitors, your own health-checker, a partner scraper) in a thin decorator checked before delegating to CrawlerDetect. Editing vendor/ is lost on the next composer update, and this package updates often.

Gotchas and Tips

Crawler ≠ bad, and detection ≠ verification. isCrawler() returns true for Googlebot, Bingbot, Slack/Discord unfurlers, Sentry's symbolicator, Pingdom, UptimeRobot, Ahrefs, and ChatGPT-User. abort(403) on all of them will deindex your site and break link previews. Keep a narrow allowlist of bots you serve, and treat the detector as a routing signal, not a security control. Anyone can spoof a UA — if you gate real content on "is Googlebot", verify by reverse DNS/ASN.

Missing UA means "not a crawler". An empty or absent User-Agent short-circuits to false. If your threat model includes bare-curl scrapers with no UA, that check is yours to add explicitly.

CLI, queues and Octane break $_SERVER. With no constructor arguments the library reads $_SERVER — meaningful only in a web SAPI. In a queued job, a console command, or an Octane worker, $_SERVER is either absent or stale from a previous request. Always pass $request->headers->all() in long-running contexts. Related: do not bind CrawlerDetect as a plain container singleton — in Octane it will serve the first request's headers forever. Use $app->scoped(...) or just new it per request.

Instantiating per request is cheap; matching is not. v1.4 memoizes the compiled regex per fixture class, so new CrawlerDetect(...) does not recompile. You still pay one large preg_match per call. Call isCrawler() once per request and store the boolean; don't call it inside a loop over rows, and don't call it from a Blade partial rendered N times.

getMatches() is a debugging aid, not a display name. It returns the matched fragment from the regex, which can look like a raw pattern or a lowercase product string. Never render it to end users without your own mapping; and always call isCrawler() first — reading the match before a check gives you the previous/empty state.

Nginx drops the underscore headers. Sec-CH-UA arrives as HTTP_SEC_CH_UA, and nginx's default underscores_in_headers off strips it. If bot detection behaves differently in production than locally, check that before you touch application code.

Don't lowercase or trim the UA yourself. The compiled patterns are already case-insensitive; normalising the string can only lose matches (and can defeat the exclusion rules).

Watch the PHP constraint. v1.4 dropped PHP 7.1 and tightened PHPUnit; if you're pinned to an old framework branch with an old PHP, Composer will resolve you to 1.3.x. Fine — just know you're getting a signature list that predates ChatGPT-User, Claude, and a batch of headless-browser detection.

Update it regularly, deliberately. New signatures land every couple of weeks. Put jaybizzle/crawler-detect in a monthly Dependabot group or a scheduled composer update — a two-year-old pin is the single most common reason "it doesn't detect bot X".

False positives are fixable without forking. If a genuine browser is being flagged, the culprit is a pattern in Crawlers.php interacting with Exclusions.php — reproduce with getMatches(), then file the UA in an issue. Recent releases are literally that: removing Flock, Amazon CloudFront, and YaBrowser over-matches.