Stop Words Laravel Package

Getting Started

Minimal Setup

composer require voku/stop-words

First Use Case: Filter stop words from a text string in English within a Laravel controller:

use Voku\StopWords\StopWords;

class SearchController extends Controller
{
    public function search(Request $request)
    {
        $query = $request->input('q');
        $stopWords = new StopWords('en');
        $filteredQuery = $stopWords->filter($query);

        // Use $filteredQuery for search logic (e.g., Scout, Algolia, or DB query)
        return SearchResult::where('content', 'like', "%{$filteredQuery}%")->get();
    }
}

Where to Look First:

• StopWords class: Core functionality for filtering.
• getAvailableLanguages(): List of supported languages (e.g., 'en', 'de', 'es').
• filter() method: Primary method for removing stop words from text.
• Laravel Service Provider: Register the package as a singleton for global access (optional).

Implementation Patterns

1. Service Integration

Pattern: Register StopWords as a Laravel singleton in a service provider.

// app/Providers/AppServiceProvider.php
use Voku\StopWords\StopWords;

public function register()
{
    $this->app->singleton(StopWords::class, function ($app) {
        return new StopWords(config('app.locale')); // Default to app locale
    });
}

Usage:

// In any controller/service
$stopWords = app(StopWords::class);
$filteredText = $stopWords->filter("The quick brown fox");

2. Middleware for Text Normalization

Pattern: Use middleware to filter stop words from incoming requests (e.g., search queries).

// app/Http/Middleware/FilterStopWords.php
public function handle($request, Closure $next)
{
    if ($request->has('q')) {
        $stopWords = app(StopWords::class);
        $request->merge(['q' => $stopWords->filter($request->input('q'))]);
    }
    return $next($request);
}

Register in app/Http/Kernel.php:

protected $middleware = [
    \App\Http\Middleware\FilterStopWords::class,
];

3. Model Observers for Content Processing

Pattern: Automatically filter stop words when saving user-generated content (e.g., blog posts).

// app/Observers/PostObserver.php
use Voku\StopWords\StopWords;

class PostObserver
{
    public function saving(Post $post)
    {
        $stopWords = new StopWords('en'); // Or inject via DI
        $post->content = $stopWords->filter($post->content);
    }
}

Register in Post model:

protected static function booted()
{
    static::observe(PostObserver::class);
}

4. Dynamic Language Selection

Pattern: Select stop-word language based on user locale or request.

// In a controller or service
$locale = app()->getLocale(); // Laravel's current locale
$stopWords = new StopWords($locale);
$filteredText = $stopWords->filter($request->input('content'));

5. Custom Stop-Word Lists per Model

Pattern: Override stop words for specific models (e.g., exclude "bank" for finance apps).

// app/Services/TextProcessor.php
use Voku\StopWords\StopWords;

class TextProcessor
{
    public function filterForModel($text, $modelClass)
    {
        $stopWords = new StopWords('en');
        if ($modelClass === Post::class) {
            $stopWords->addStopWords(['custom', 'terms']); // Model-specific exclusions
        }
        return $stopWords->filter($text);
    }
}

6. Queue Jobs for Batch Processing

Pattern: Process large datasets asynchronously (e.g., cleaning old comments).

// app/Jobs/ProcessCommentsJob.php
use Voku\StopWords\StopWords;

class ProcessCommentsJob implements ShouldQueue
{
    public function handle()
    {
        $stopWords = new StopWords('en');
        Comment::chunk(100, function ($comments) use ($stopWords) {
            foreach ($comments as $comment) {
                $comment->update([
                    'content' => $stopWords->filter($comment->content)
                ]);
            }
        });
    }
}

Dispatch the job:

ProcessCommentsJob::dispatch();

7. Integration with Laravel Scout

Pattern: Filter stop words before indexing searchable content.

// app/Models/Post.php
use Laravel\Scout\Searchable;

class Post extends Model
{
    use Searchable;

    public function toSearchableArray()
    {
        $stopWords = app(StopWords::class);
        return [
            'title' => $stopWords->filter($this->title),
            'content' => $stopWords->filter($this->content),
        ];
    }
}

8. API Responses

Pattern: Filter stop words from API responses (e.g., search results).

// app/Http/Controllers/SearchController.php
public function index(Request $request)
{
    $results = SearchModel::search($request->input('q'));
    $stopWords = app(StopWords::class);

    return response()->json(
        array_map(function ($result) use ($stopWords) {
            return [
                'title' => $stopWords->filter($result->title),
                'excerpt' => $stopWords->filter($result->excerpt),
            ];
        }, $results)
    );
}

Gotchas and Tips

Pitfalls

1. Case Sensitivity:

   • Stop-word lists are case-sensitive by default. Convert text to lowercase first if needed:

     $stopWords->filter(strtolower($text));
     

2. Punctuation Handling:

   • The package does not strip punctuation. Use Str::of($text)->lower()->words() or preg_replace to clean text first:

     use Illuminate\Support\Str;
     $cleanText = Str::of($text)->lower()->words()->implode(' ');
     $filtered = $stopWords->filter($cleanText);
     

3. Language Mismatches:

   • If the input text contains mixed languages, filtering may produce unexpected results. Validate language or pre-process text:

     if (!in_array($locale, $stopWords->getAvailableLanguages())) {
         throw new \InvalidArgumentException("Unsupported language: {$locale}");
     }
     

4. Performance with Large Texts:

   • Processing very long texts (e.g., >10KB) may cause timeouts. Chunk the text or use async processing:

     $chunkSize = 5000;
     $chunks = array_chunk(str_word_count($text, 1), $chunkSize);
     $filteredChunks = array_map(fn($chunk) => $stopWords->filter(implode(' ', $chunk)), $chunks);
     $filteredText = implode(' ', $filteredChunks);
     

5. Default Language Fallback:

   • If no language is specified, the package defaults to English. Explicitly set the language to avoid surprises:

     $stopWords = new StopWords('en'); // Always specify
     

Debugging Tips

1. Inspect Stop-Word Lists:

   • Log the stop words for a language to verify coverage:

     logger()->debug('Stop words for English:', $stopWords->getStopWords('en'));
     

2. Test Edge Cases:

   • Test with empty strings, special characters, and mixed-language text:

     $stopWords->filter(""); // Should return ""
     $stopWords->filter("Café au lait"); // May not handle non-Latin scripts well
     

3. Validate Filtered Output:

   • Ensure no critical terms are accidentally filtered (e.g., "bank" in finance apps):

     $filtered = $stopWords->filter("The bank is open");
     $this->assertNotContains("bank", $filtered); // If "bank" should not be filtered
     

4. Memory Usage:

   • Monitor memory consumption for batch processing:

     $memoryBefore = memory_get_usage();
     $filtered = $stopWords->filter($largeText);
     $memoryAfter = memory_get_usage();
     logger()->debug("Memory used: " . ($memoryAfter - $memoryBefore) . " bytes");
     

Configuration Quirks

1. Language Codes:

   • Use ISO 639-1 codes (e.g., 'en' for English, 'de' for German). Check getAvailableLanguages() for the full list.

2. Custom Stop-Word Files:

   • The package does not support loading stop words from external files. Override lists programmatically:

     $stopWords = new StopWords('en');
     $stopWords->setStopWords(['custom', 'list', 'here']);
     

3. Caching:

   • Stop-word lists are static