Detect Encoding Laravel Package

Getting Started

Install via Composer:

composer require onnov/detect-encoding

First use case: Replace unreliable mb_detect_encoding for Cyrillic/legacy text in a Laravel controller:

use Onnov\DetectEncoding\EncodingDetector;

class TextController extends Controller
{
    public function processLegacyText(string $rawText)
    {
        $detector = new EncodingDetector();
        $encoding = $detector->getEncoding($rawText);

        // Convert to UTF-8 for storage
        $utf8Text = $detector->iconvXtoEncoding($rawText);

        // Store or process...
    }
}

Key starting points:

1. README accuracy table to validate coverage for your encodings.
2. Test examples for edge cases.
3. EncodingDetector::class constants (e.g., IBM866, WINDOWS_1251) for configuration.

Implementation Patterns

1. Service Integration

Wrap in a Laravel service with dependency injection:

// app/Services/EncodingService.php
class EncodingService
{
    protected $detector;

    public function __construct(EncodingDetector $detector)
    {
        $this->detector = $detector;
        $this->configureDetector();
    }

    protected function configureDetector(): void
    {
        $this->detector->enableEncoding([
            EncodingDetector::WINDOWS_1251,
            EncodingDetector::KOI8_R,
        ]);
    }

    public function detectAndConvert(string $text): string
    {
        return $this->detector->iconvXtoEncoding($text);
    }
}

Register in AppServiceProvider:

public function register()
{
    $this->app->singleton(EncodingService::class, function ($app) {
        return new EncodingService(new EncodingDetector());
    });
}

2. Middleware for Incoming Text

Detect encoding in API requests or file uploads:

// app/Http/Middleware/DetectEncoding.php
public function handle($request, Closure $next)
{
    if ($request->has('legacy_text')) {
        $detector = app(EncodingDetector::class);
        $encoding = $detector->getEncoding($request->legacy_text);
        $request->merge(['detected_encoding' => $encoding]);
    }
    return $next($request);
}

3. Artisan Command for Bulk Processing

php artisan encoding:convert database/legacy_texts.csv

Command implementation:

protected $signature = 'encoding:convert {file}';
protected $detector;

public function __construct(EncodingDetector $detector)
{
    $this->detector = $detector;
}

public function handle()
{
    $contents = file_get_contents($this->argument('file'));
    $utf8Text = $this->detector->iconvXtoEncoding($contents);
    file_put_contents("converted_{$this->argument('file')}", $utf8Text);
}

4. Event Listeners

Trigger on file uploads or imports:

// app/Listeners/DetectUploadEncoding.php
public function handle(FileUploaded $event)
{
    $detector = app(EncodingDetector::class);
    $encoding = $detector->getEncoding($event->file->getContent());
    // Log or process encoding metadata
}

5. Configuration-Driven Workflows

Use Laravel config to manage enabled/disabled encodings:

// config/encoding.php
return [
    'enabled_encodings' => [
        EncodingDetector::WINDOWS_1251,
        EncodingDetector::KOI8_R,
        EncodingDetector::ISO_8859_5,
    ],
    'default_target_encoding' => 'utf-8',
    'iconv_options' => ['//TRANSLIT'],
];

Service initialization:

public function __construct(EncodingDetector $detector)
{
    $this->detector = $detector;
    $this->detector->enableEncoding(config('encoding.enabled_encodings'));
}

Gotchas and Tips

Pitfalls

1. Short Text Accuracy:

   • Accuracy drops below 90% for texts <15 characters (per README). Tip: Pad short texts with whitespace or use a fallback for critical data.
   • Example:

     $text = str_pad($shortText, 30, ' ');
     $encoding = $detector->getEncoding($text);
     

2. Windows-1251 vs. KOI8-R Confusion:

   • These encodings share overlapping character ranges, leading to false positives. Tip: Use domain knowledge (e.g., if text is from a Russian Windows app, prioritize WINDOWS_1251).

3. MacCyrillic/IBM866 Disabled by Default:

   • These are disabled in the default config (low accuracy for short texts). Tip: Enable only if explicitly needed:

     $detector->enableEncoding([EncodingDetector::IBM866]);
     

4. Iconv Limitations:

   • iconvXtoEncoding() relies on PHP’s iconv, which may fail for unsupported encodings. Tip: Validate target encoding support first:

     if (!in_array($targetEncoding, iconv_encodings())) {
         throw new \RuntimeException("Unsupported target encoding: {$targetEncoding}");
     }
     

5. Mixed Encoding Text:

   • The detector assumes uniform encoding. Tip: Pre-process text to remove mixed-content warnings or use a fallback for ambiguous cases.

6. Performance with Large Texts:

   • While fast for detection, iconvXtoEncoding() can be slow for multi-MB files. Tip: Process in chunks or use streaming:

     $handle = fopen($largeFile, 'r');
     while (!feof($handle)) {
         $chunk = fread($handle, 8192);
         $utf8Chunk = $detector->iconvXtoEncoding($chunk);
         // Process chunk...
     }
     

Debugging Tips

1. Verify Encodings:

   • Log detected encodings to validate accuracy:

     $encoding = $detector->getEncoding($text);
     \Log::debug("Detected encoding for sample text: {$encoding}");
     

2. Test Edge Cases:

   • Use the test suite to validate:
     • Short texts (<15 chars).
     • Texts with punctuation/symbols.
     • Mixed-case Cyrillic.

3. Fallback Strategy:

   • Combine with mb_detect_encoding for broader coverage:

     $detector = new EncodingDetector();
     $encodings = [$detector->getEncoding($text)];
     if (empty($encodings)) {
         $encodings = ['UTF-8', 'WINDOWS-1251', 'KOI8-R'];
     }
     $encoding = mb_detect_encoding($text, implode(',', $encodings));
     

4. Custom Encoding Validation:

   • If adding custom encodings, validate ranges with:

     $codePage = new \Onnov\DetectEncoding\CodePage();
     $range = $codePage->getRange($uppercase, $lowercase, 'custom_name');
     

Extension Points

1. Add Custom Encodings:

   • Extend for unsupported encodings (e.g., Big5, Shift_JIS):

     $detector->addEncoding([
         'big5' => [
             'upper' => '0xA1-0xFE', // Example range (research required)
             'lower' => '0x40-0x7E',
         ],
     ]);
     

2. Create a Facade:

   • Laravel-specific wrapper for cleaner syntax:

     // app/Facades/Encoding.php
     public static function detect(string $text): string
     {
         return app(EncodingDetector::class)->getEncoding($text);
     }
     

3. Integrate with Laravel Filesystem:

   • Auto-detect encoding for uploaded files:

     $file = $request->file('legacy_file');
     $contents = $file->get();
     $encoding = Encoding::detect($contents);
     $utf8Contents = Encoding::convert($contents, 'utf-8');
     

4. Cache Results:

   • Cache detection for repeated texts (e.g., API responses):

     $cacheKey = 'encoding_'.md5($text);
     $encoding = Cache::remember($cacheKey, now()->addHours(1), function() use ($text) {
         return $detector->getEncoding($text);
     });
     

Configuration Quirks

1. Default Behavior:
   • iconvXtoEncoding() defaults to utf-8 with //TRANSLIT. Over