Matcher Laravel Package

Getting Started

Minimal Setup

1. Install the package:

   composer require loupe/matcher
   

2. Basic usage in a Laravel controller:

   use Loupe\Matcher\Tokenizer\Tokenizer;
   use Loupe\Matcher\Matcher;
   use Loupe\Matcher\Formatter;
   use Loupe\Matcher\FormatterOptions;
   
   public function search(Request $request)
   {
       $tokenizer = new Tokenizer();
       $matcher = new Matcher($tokenizer);
       $formatter = new Formatter($matcher);
   
       $query = $request->input('q');
       $text = "Your long document text here...";
   
       $options = (new FormatterOptions())
           ->withEnableHighlight()
           ->withHighlightStartTag('<mark>')
           ->withHighlightEndTag('</mark>')
           ->withEnableCrop()
           ->withCropLength(100);
   
       $result = $formatter->format($text, $query, $options);
       return response()->json(['snippet' => $result->getFormattedText()]);
   }
   

First Use Case: Search Result Snippets

For a Laravel search feature (e.g., Algolia/Scout results), use this to generate snippets:

// In your search result view or Livewire component
$snippet = $formatter->format($product->description, $request->query, $options);
echo $snippet->getFormattedText();

Where to Look First

• Core classes: Tokenizer, Matcher, Formatter (in src/).
• Configuration: FormatterOptions for customizing output.
• Tests: tests/ for usage patterns (e.g., FormatterTest.php for cropping logic).

Implementation Patterns

Workflow: Search Snippet Generation

1. Tokenize the query:

   $tokenizer = new Tokenizer('en_US');
   $queryTokens = $tokenizer->tokenize($request->query);
   

2. Match against text:

   $matcher = new Matcher($tokenizer, ['the', 'and']);
   $matches = $matcher->calculateMatches($documentText, $queryTokens);
   

3. Format with highlights and cropping:

   $options = (new FormatterOptions())
       ->withEnableHighlight()
       ->withEnableCrop()
       ->withCropLength(150);
   $result = $formatter->format($documentText, $queryTokens, $options);
   

Integration with Laravel Scout

// In your Scout model
public function toSearchableArray()
{
    return [
        'title' => $this->title,
        'description' => $this->description,
        'snippet' => function () {
            $tokenizer = new Tokenizer();
            $matcher = new Matcher($tokenizer);
            $formatter = new Formatter($matcher);
            return $formatter->format($this->description, $this->searchQuery, $options);
        }
    ];
}

Livewire Component Example

// app/Http/Livewire/SearchResults.php
public function render()
{
    $snippets = collect($this->results)
        ->map(fn ($result) => $this->generateSnippet($result->text, $this->query));

    return view('livewire.search-results', ['snippets' => $snippets]);
}

protected function generateSnippet(string $text, string $query): string
{
    $formatter = resolve(Formatter::class);
    $options = (new FormatterOptions())
        ->withEnableHighlight()
        ->withCropLength(200);
    return $formatter->format($text, $query, $options)->getFormattedText();
}

Caching Matches for Performance

// Cache matches for 1 hour
$cacheKey = "matches:{$documentId}:{$query}";
$matches = Cache::remember($cacheKey, now()->addHours(1), function () use ($document, $query) {
    $tokenizer = new Tokenizer();
    $matcher = new Matcher($tokenizer);
    return $matcher->calculateMatches($document->text, $query);
});

// Use cached matches in formatter
$result = $formatter->format($document->text, $query, $options, $matches);

Custom Tokenizer for Domain-Specific Needs

// Example: Tokenizer for product attributes (e.g., "color:red", "size:large")
class ProductAttributeTokenizer implements TokenizerInterface
{
    public function tokenize(string $text): TokenCollection
    {
        preg_match_all('/([a-z]+):([a-z]+)/i', $text, $matches);
        return new TokenCollection(array_map(
            fn($attr, $value) => new Token("{$attr}:{$value}"),
            $matches[1],
            $matches[2]
        ));
    }

    public function matches(Token $token, TokenCollection $tokens): bool
    {
        return $tokens->contains(fn($t) => $t->getValue() === $token->getValue());
    }
}

Gotchas and Tips

Pitfalls

1. Locale-Specific Tokenization:

   • Always specify a locale (e.g., 'en_US') for accurate word boundaries. Defaults to C (ASCII), which may split words incorrectly (e.g., "lorem ipsum" → ["lorem", "ipsum"] vs. ["lorem ipsum"]).
   • Fix: Pass the locale to the Tokenizer:

     $tokenizer = new Tokenizer('en_US');
     

2. Phrase Matching Quirks:

   • Quoted phrases ("exact phrase") must match exactly, including case and punctuation. Use lowercase: true in FormatterOptions for case-insensitive matching:

     $options->withLowercase(true);
     

   • Gotcha: Negated terms (-exclude) are treated as stop words for the entire query. Avoid mixing them with other terms if unintended exclusions occur.

3. HTML/Tag Handling:

   • The formatter does not parse HTML. If your text contains tags (e.g., <b>bold</b>), they’ll be treated as literal text. For HTML-aware cropping, use the standalone Cropper:

     $cropper = new \Loupe\Matcher\Formatting\Cropper(
         cropLength: 100,
         highlightStartTag: '<mark>',
         highlightEndTag: '</mark>'
     );
     $cropped = $cropper->cropHighlightedText($htmlText);
     

4. Performance with Long Texts:

   • Tokenizing and matching large documents (e.g., 10KB+) can be slow. Pre-calculate matches and cache them:

     $matches = Cache::remember("matches:{$documentId}", now()->addDays(1), function () {
         return $matcher->calculateMatches($longDocumentText, $query);
     });
     

5. Diacritics and Unicode:

   • The library uses ICU for diacritic normalization (e.g., é → e), but ensure ext-intl is enabled in your PHP config (php.ini):

     extension=intl
     

   • Test: Verify with accented terms like "café" or "naïve".

6. Formatter Options Overrides:

   • Options are immutable. To modify them, create a new instance:

     // Wrong: This won't work
     $options->withCropLength(200)->withHighlightStartTag('<span>');
     
     // Correct: Chain or create new
     $newOptions = $options->withCropLength(200);
     

Debugging Tips

1. Inspect Tokens:

   $tokens = $tokenizer->tokenize($text);
   dd($tokens->all()); // Debug tokenized output
   

2. Match Span Visualization:

   $spans = $matcher->calculateMatchSpans($text, $query, $matches);
   foreach ($spans as $span) {
       echo "Match at {$span->getStartPosition()}-{$span->getEndPosition()}: "
           . substr($text, $span->getStartPosition(), $span->getLength()) . "\n";
   }
   

3. Disable Cropping for Testing:

   $options = (new FormatterOptions())->withEnableCrop(false);
   

Extension Points

1. Custom Highlight Tags:

   $options->withHighlightStartTag('<span class="highlight">')
           ->withHighlightEndTag('</span>');
   

2. Post-Processing Formatted Text:
   • Use Result::getFormattedText() and apply additional transformations (e.g., strip tags, sanitize):

     $formatted = $result->getFormattedText();
     $sanitized = strip_tags($formatted, '<mark><em>');
     

3. Event Hooks (Advanced):
   • Extend Formatter to add callbacks for pre/post-formatting:

     class CustomFormatter extends Formatter
     {
         protected function beforeFormat(string $text, TokenCollection $query