Laravel Crawler Detect Laravel Package

Getting Started

Install

composer require jaybizzle/laravel-crawler-detect

The service provider and Crawler facade alias are auto-discovered on Laravel 5.5+. For Lumen or older apps, register Jaybizzle\LaravelCrawlerDetect\LaravelCrawlerDetectServiceProvider in bootstrap/app.php and alias Jaybizzle\LaravelCrawlerDetect\Facades\Crawler in config/app.php.

Look here first

1. The facade: Jaybizzle\LaravelCrawlerDetect\Facades\Crawler — it proxies to a singleton Jaybizzle\CrawlerDetect\CrawlerDetect.
2. The underlying library's regex list (CrawlerDetect::$crawlers). That array is where every detection decision comes from; the Laravel package is a thin container binding around it.
3. Any published config — run php artisan vendor:publish and check what lands, because available knobs differ between wrapper versions.

Smallest useful call

use Jaybizzle\LaravelCrawlerDetect\Facades\Crawler;

if (Crawler::isCrawler()) {
    // current request's User-Agent matched a known bot
}

First real use case — a global middleware that labels bot traffic so it never reaches session/cookie-heavy logic, or an analytics listener that tags bot hits instead of counting them as human.

Note that isCrawler() with no argument reads the current request's User-Agent from the server globals. Anywhere you don't have a live request (queues, commands, tests, php artisan tinker) you must pass the UA explicitly:

Crawler::isCrawler($request->userAgent());

Implementation Patterns

Middleware gate (most common)

class BlockCrawlers
{
    public function handle($request, Closure $next)
    {
        if (Crawler::isCrawler($request->userAgent())) {
            abort(403);
        }

        return $next($request);
    }
}

Register it globally or at the top of a route group rather than inside the web group if you want to skip session and CSRF work for bots entirely.

Tag, don't just block — many teams want bot traffic recorded but not processed. Attach the verdict to the request and read it later:

$request->attributes->set('is_crawler', Crawler::isCrawler($request->userAgent()));

Then check that attribute in logging, analytics listeners, or response middleware. This avoids re-running the (large) regex set multiple times per request.

SEO / prerender split — serve prerendered or simplified HTML to crawlers, full SPA shell to humans:

if (Crawler::isCrawler($request->userAgent())) {
    return response()->view('prerender.page', $data);
}

Debugging a detection — when you need to know which rule fired:

Crawler::isCrawler($ua);
$matches = Crawler::getMatches(); // names/patterns that matched

Log both the verdict and the matches when investigating false positives.

Custom rules and exclusions — the wrapper exposes the underlying instance, so use its setters rather than forking the package:

Crawler::setCrawlers(array_merge(Crawler::getCrawlers(), [
    '/MyInternalMonitor\/\d+/i',
]));

Crawler::setExclusions(array_merge(Crawler::getExclusions(), [
    '/SomeLegitReader/i',
]));

Do this in a service provider's boot() so every resolution of the singleton sees the additions.

Testing — never rely on the test runner's $_SERVER; always pass the UA string:

$this->assertTrue(Crawler::isCrawler('Mozilla/5.0 (compatible; Googlebot/2.1)'));
$this->assertFalse(Crawler::isCrawler('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ...'));

Queues and scheduled jobs — pass the UA through the job payload from the request that enqueued it, or store it alongside the record. There is no request context in a worker.

Gotchas and Tips

• No request context means no detection. In CLI, queue workers, and tests the server globals are absent, so isCrawler() returns false silently. Always pass $request->userAgent() explicitly and make the parameter part of your code review checklist.
• It's User-Agent based, not identity based. Googlebot is trivially spoofable. For anything security-sensitive (serving privileged or canonical content), verify the crawler's published IP ranges or do reverse DNS — the regex package only tells you what the client claims to be.
• Cloaking risk. Serving materially different content to detected crawlers can be treated as cloaking by search engines. Keep crawler-specific responses equivalent in substance, or restrict the split to prerendering the same content.
• False positives are real. Headless browsers, feed readers, monitoring/Uptime tools, link previewers, and some corporate proxies match. Use the exclusion list for known-good clients, and always log getMatches() output when a user reports being blocked so you can identify the exact pattern.
• Empty or missing UA. A blank User-Agent doesn't reliably match the crawler regexes. Decide explicitly whether a missing UA should be treated as a bot (usually yes, for abuse control) rather than assuming the package handles it.
• Performance. The pattern set is thousands of alternated regexes. Calling isCrawler() on every request, in every listener, and again in a view composer adds up. Resolve once per request and reuse the result.
• Singleton state. The facade resolves a shared instance. If the wrapper captures the User-Agent at construction time, long-lived workers (Octane, Swoole, queue daemons) may see a stale UA on subsequent requests. Pass the UA explicitly per call, or resolve a fresh CrawlerDetect instance, rather than trusting the singleton to re-read the request.
• Customizations must happen early. Patterns added in a controller or middleware after the first isCrawler() call may not apply the way you expect if the instance has already cached a result. Register additions in a service provider.
• Pattern freshness matters. New bots appear constantly; the detection quality is entirely the underlying jaybizzle/crawler-detect release. Keep the dependency updated and don't hand-edit the shipped regex list — add your own patterns on top instead.
• Middleware ordering. If the goal is to avoid creating sessions for bots, the crawler check must run before StartSession and VerifyCsrfToken. Dropping it into the web group puts it too late to save that work.
• Proxies and IPs. The UA header survives proxies, but if you also block by IP, make sure TrustProxies is configured or $request->ip() will report your load balancer for every bot.
• Facade alias collisions. The Crawler alias is generic; if another package claims it, alias differently in config/app.php and import the FQCN in your code.
• Wrapper is thin — don't expect config magic. Anything beyond the basic facade calls lives in the underlying CrawlerDetect class. When the wrapper's docs are thin, read the upstream library's source for setCrawlers, setExclusions, getMatches, and header handling.