Efficient Language Detector Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require nitotm/efficient-language-detector
   

   Use --prefer-dist to exclude tests/benchmarks or --prefer-source to include them.

2. First Detection:

   use Nitotm\Eld\LanguageDetector;
   
   $detector = new LanguageDetector(); // Default: 'large' size, 'string' mode
   $result = $detector->detect('Bonjour, comment ça va?');
   echo $result->language; // Outputs: 'fr'
   

3. Key Properties:

   • language: ISO 639-1 code (e.g., 'fr').
   • scores(): Associative array of language probabilities (e.g., ['fr' => 0.95, 'en' => 0.03]).
   • isReliable(): Boolean indicating confidence in detection.

Implementation Patterns

Core Workflows

1. Basic Detection:

   $detector = new LanguageDetector();
   $text = "Hola, ¿cómo estás?";
   $detection = $detector->detect($text);
   if ($detection->isReliable()) {
       // Proceed with $detection->language
   }
   

2. Language Subsets:

   • Dynamic Subset (for string, bytes, disk modes):

     $detector->langSubset(['en', 'es', 'fr']); // Limits detection to these languages
     

   • Pre-built Subset (for array mode):

     $subsetFile = $detector->langSubset(['en', 'es'], save: true);
     $subsetDetector = new LanguageDetector($subsetFile, null, 'array');
     

3. Output Schemes:

   $detector->setOutputScheme('ISO639_2T'); // Changes output to ISO 639-2/T
   

4. Text Cleanup (Optional):

   $detector->enableTextCleanup(true); // Removes URLs, emails, etc. (may reduce accuracy)
   

Integration Tips

1. Batch Processing:

   $texts = ["Bonjour", "Hallo", "こんにちは"];
   $results = array_map(fn($text) => $detector->detect($text), $texts);
   

2. Caching Detector Instances:

   • Reuse the same instance for multiple detections (avoids reloading databases).
   • For array mode, enable OPcache to optimize performance:

     opcache.interned_strings_buffer=128
     opcache.memory_consumption=256
     

3. Database Mode Selection:

   • array: Fastest (requires OPcache), highest memory usage.
   • string/bytes: Balanced speed/memory (2x slower than array).
   • disk: Lowest memory (slower, but scalable for large datasets).

4. CLI Usage:

   php bin/eld "Bonjour"  # Returns detected language
   

Gotchas and Tips

Pitfalls

1. Memory Limits:

   • array mode can consume significant memory (e.g., extralarge may need 1GB+).
   • Use string/bytes modes for memory-constrained environments (e.g., shared hosting).

2. Text Encoding:

   • Input must be UTF-8. Non-UTF-8 text may yield incorrect results.
   • Validate input with mb_detect_encoding() if unsure:

     if (mb_detect_encoding($text) !== 'UTF-8') {
         $text = mb_convert_encoding($text, 'UTF-8');
     }
     

3. Subset Overhead:

   • In array mode, the first langSubset() call is slow (builds a new database). Save subsets for reuse:

     $subsetFile = $detector->langSubset(['en', 'es'], save: true);
     

4. Text Cleanup Tradeoff:

   • Enabling enableTextCleanup(true) removes URLs/emails, which may contain language hints (e.g., .com.br suggests Portuguese). Use sparingly.

5. Disk Mode Latency:

   • disk mode is slow for single detections but ideal for batch processing (e.g., processing 1000+ texts).

Debugging Tips

1. Check Reliability:

   if (!$detection->isReliable()) {
       log::warning("Unreliable detection for text: {$text}");
   }
   

2. Inspect Scores:

   • Low confidence in top result? Check scores() for close competitors:

     $scores = $detection->scores();
     $topScore = max($scores);
     if ($topScore < 0.7) {
         // Handle ambiguous cases (e.g., fallback to user input or default language)
     }
     

3. Database Info:

   $info = $detector->info();
   // Debug supported languages, mode, or memory usage
   

4. Benchmarking:

   • Compare modes/sizes using the benchmarks table.
   • Example: extralarge in disk mode uses ~0.5MB RAM but is 10x slower than array mode.

Extension Points

1. Custom Language Models:

   • Train on domain-specific text (e.g., legal, medical) by extending the database. See BlobDataBuilder for custom builds.

2. Fallback Logic:

   $fallbackLanguages = ['en', 'es'];
   $detection = $detector->detect($text);
   $language = in_array($detection->language, $fallbackLanguages)
       ? $detection->language
       : $fallbackLanguages[0];
   

3. Event Listeners:

   • Hook into detection results for analytics or logging:

     $detector->detect($text); // Trigger custom logic via events
     

4. CLI Automation:

   • Use the CLI wrapper for scheduled tasks (e.g., batch-processing logs):

     php bin/eld "input.txt" > output.json