Forceutf8 Laravel Package

Getting Started

Install

composer require fylax/forceutf8

There is no service provider, no config file, and nothing to publish. The package is a single namespace-level utility class. If you're looking for config/forceutf8.php, it doesn't exist — don't waste time on php artisan vendor:publish.

Where to look first

• ForceUTF8\Encoding — the whole API lives here as static methods.
• Encoding::toUTF8($string) — the workhorse; converts a mixed-encoded string to UTF-8 by detecting valid UTF-8 sequences and converting the rest (assuming Windows-1252/ISO-8859-1) via a precomputed conversion map.
• Encoding::toUTF8Array(array $array) — recursively applies toUTF8 to every scalar in an array.
• Encoding::fixUTF8($string) — for mojibake (double-encoded strings like ñ instead of ñ). This is a different problem from mixed encoding; read the docblock before using it.
• Encoding::isUTF8($string) — cheap boolean check.
• Encoding::utf8badge($string) — returns 'UTF-8' or 'ISO-8859-1'; handy for logging what was actually detected.

Minimal first use case

The classic entry point is a legacy CSV or an old MySQL dump that came back as "José" / "François":

use ForceUTF8\Encoding;

$row = array_map(fn ($v) => Encoding::toUTF8((string) $v), fgetcsv($handle));

Or, if you want the recursion for free:

Encoding::toUTF8Array($row);

Verify against a fixture with known mojibake before you trust it in a import command — heuristic encoding detection is not a solved problem, and this class is a heuristic.

Implementation Patterns

1. Sanitize at the boundary, not in Blade

Encoding repair is a write-side concern. Normalize on ingress (imports, webhooks, API payloads, form submissions) so the database is UTF-8 clean. Don't sprinkle Encoding::toUTF8() into Blade templates — it will convert already-valid strings on every render for no benefit.

2. FormRequest prepareForValidation

Most user-facing need is someone pasting text from Word/Excel (Windows-1252 smart quotes, en-dashes) into a rich field:

protected function prepareForValidation(): void
{
    $clean = [];
    foreach ($this->all() as $key => $value) {
        $clean[$key] = is_string($value) ? Encoding::toUTF8($value) : $value;
    }
    $this->replace($clean);
}

Apply it on the specific request classes that accept free text, not globally in middleware — you don't want to touch file uploads or signed payload hashes.

3. Eloquent mutators for known-bad legacy columns

For a table fed by a third-party integration that you can't trust:

protected function legacyNotes(): Attribute
{
    return Attribute::make(
        set: fn (?string $value) => $value === null ? null : Encoding::toUTF8($value),
    );
}

Keep this narrow. A blanket trait on the base model will silently rewrite data you didn't intend to touch and makes debugging harder.

4. Bulk backfill as an Artisan command

To repair existing rows, wrap toUTF8 in a chunked command rather than an update-all:

Customer::query()
    ->whereNotNull('name')
    ->chunkById(500, function ($customers) {
        foreach ($customers as $customer) {
            if (! Encoding::isUTF8($customer->name)) {
                $customer->name = Encoding::toUTF8($customer->name);
                $customer->saveQuietly();
            }
        }
    });

Log the diff (or count) so you have an audit trail — you're rewriting production text.

5. Import pipelines

For CSV/TSV ingestion, toUTF8Array on each row is convenient, but note it walks every key including nested arrays. For large files, apply toUTF8 per field inside your existing row loop and skip the recursion overhead. Combine with a BOM strip (see Gotchas).

6. Normalization after conversion

Encoding and Unicode normalization are separate problems. After toUTF8, if you're storing slugs, searchable names, or building matching logic, follow up with:

$utf8 = Encoding::toUTF8($raw);
$nfkc = \Normalizer::normalize($utf8, \Normalizer::FORM_C); // requires ext-intl
$slug = \Illuminate\Support\Str::ascii($nfkc);

Str::ascii() alone won't fix mojibake — it will happily turn ñ into A+/- nonsense.

7. Queue and webhook payloads

When decoding JSON from a third-party webhook, decode first, then pass strings through toUTF8. Never run it on the raw JSON body — you'll corrupt the JSON structure if the payload contains mixed byte sequences. For job payloads from external systems, sanitize inside handle() before persisting.

8. Tests

Keep a fixture file of intentionally malformed byte sequences (Windows-1252 \xE9 for é, \x93/\x94 curly quotes, and a double-encoded é). Assert both toUTF8 and fixUTF8 behavior so an upgrade that changes the conversion map is caught immediately. This package is a heuristic — pin the version and test the upgrade.

Gotchas and Tips

It fixes mixed encoding, not arbitrary encodings. toUTF8() assumes the non-UTF-8 bytes are Windows-1252/ISO-8859-1. Shift-JIS, GBK, Big5, KOI8-R, and other multi-byte charsets are not handled. For those you need mb_convert_encoding($s, 'UTF-8', 'Shift-JIS') with the encoding known up front, or mb_detect_encoding as a starting guess.

Mojibake is a different function. If text looks like é, ’, or “, the bytes are valid UTF-8 — toUTF8() will do nothing. You need fixUTF8(), and even then the result depends on which side of the double-encoding went wrong. Don't assume one function covers both; test both against real samples from your data.

isUTF8() is a regex check, not a validator. It's permissive and fast. It is not equivalent to mb_check_encoding($s, 'UTF-8') for strict validation. If you're using it as a security gate, use mb_check_encoding instead.

Strip the BOM yourself. UTF-8 BOM (\xEF\xBB\xBF) at the start of CSV/JSON often survives conversion and breaks json_decode and the first column of imports. Use Str::of($s)->replaceStart("\xEF\xBB\xBF", '') or preg_replace('/^\x{FEFF}/u', '', $s) before/after conversion.

Never run it on binary data. Uploaded images, PDFs, or zipped payloads will be mangled byte-for-byte by the conversion map. Guard with a MIME check, or exclude file fields from any blanket toUTF8Array() on $request->all().

toUTF8Array() only understands arrays. It will not recurse into Collection, Arrayable, Jsonable, stdClass, or ArrayObject instances. Convert to array first ($collection->toArray()) or map over the collection manually — otherwise objects pass through untouched and you get silent inconsistency between array and object code paths.

Performance on large payloads. The conversion map and per-character scanning mean this is noticeably slower than mb_convert_encoding. On a 10 MB log file or a 50k-row import, budget for it: process in chunks, and skip the call entirely if isUTF8() returns true. Don't put it in a hot loop over already-clean data.

It's not a sanitizer. toUTF8() does not strip control characters, nulls, or invalid XML characters, and it does not prevent XSS. Keep escaping with Blade's {{ }} and keep strip_tags / HTML sanitizers separate. Treat encoding repair as a data-integrity step only.

Database collation still matters. Converting strings to UTF-8 in PHP is pointless if the column is utf8 (3-byte) and the user pastes a 4-byte emoji. Use utf8mb4 with utf8mb4_unicode_ci in config/database.php, and ensure the connection charset is utf8mb4. This package cannot compensate for a misconfigured connection.

Extension dependencies. The conversion path relies on mbstring (and in some branches iconv). Verify both are present in production — php -m | grep -E 'mbstring|iconv' — before deploying code that calls it in a request path.

Over-conversion risk. The heuristic occasionally rewrites bytes that were already valid in a non-UTF-8 context you didn't anticipate. Always keep the original value alongside the converted one during a backfill (e.g. a *_raw column or a dump) so you can roll back if a batch of names comes out worse than it went in.

Version and maintenance. This is a small fork (24 stars, last release 2026-10-25) of the older neitanod/forceutf8. There's no Laravel integration, no config, and no extension points beyond subclassing Encoding or wrapping it in your own service. Pin the exact version in composer.json, wrap the static calls behind your own small class (e.g. App\Support\Utf8Fixer) so you can swap the implementation without touching every call site, and check the upstream diff before upgrading — conversion-map changes can alter output for edge-case inputs.