Json Schema Test Suite Laravel Package

Technical Evaluation

Architecture fit. This is not a library — it is a language-agnostic corpus of JSON documents (schemas, instances, and expected-validity flags) plus a small Python helper (bin/jsonschema_suite) for materializing remote references. There is no PHP code, no autoloader, no service provider, no runtime surface. It can only ever sit at the test/CI layer of a Laravel application. That makes the fit binary: near-zero value for a typical CRUD app whose only contact with JSON Schema is Laravel's json validation rule, and high value for a team that (a) is selecting or upgrading a PHP JSON Schema validator, (b) maintains custom keywords/formats on top of one, or (c) generates schemas from Laravel constructs (form requests, spatie/laravel-data, OpenAPI output) and needs a conformance baseline. The low opportunity score reflects that narrowness, not a quality problem — the suite is the de facto reference implementation of the spec's behavior.

Integration feasibility. No Composer package is published by upstream. Consumption options, in order of preference: a pinned commit referenced as a type: library VCS repository in composer.json; a CI download step that curls a tarball at a fixed SHA; or a git submodule. All three require the team to write the runner — roughly 200–400 lines to walk the draft directories, emit PHPUnit/Pest data providers, and supply a custom URI retriever that maps http://localhost:1234/... remote refs to the local remotes/ tree. That runner is the actual engineering deliverable; the suite is the fixture set.

Technical risk. Four concrete hazards. (1) Draft mismatch: the corpus spans draft4 through 2020-12, but PHP validators are uneven — justinrainbow/json-schema is draft-4-centric, swaggest/php-json-schema is draft-7, opis/json-schema covers 6/7/2019-09/2020-12. Running drafts your validator doesn't claim to support generates noise, not signal. (2) No upstream semver tags; last release May 2023. Pin to a SHA and treat it as frozen, because a silent bump can flip expected outcomes after upstream test corrections. (3) Remote-reference tests fail in offline CI unless the retriever is implemented; the Python helper is a workaround, not an integration. (4) A non-trivial number of cases are known-ambiguous (float precision, regex dialect differences, format assertion semantics), so a justified skip-list is mandatory — and it becomes the primary operational liability if unmanaged.

Key questions. Which draft(s) do our schemas actually declare? Do we need format as assertion or annotation only (the latter needs no assertion libraries)? Does our validator expose a pluggable URI retriever and standard output format, or only a boolean? Are we evaluating a vendor library or regression-testing our own extensions? Who owns the skip-list and its written rationale? Is conformance a release gate or a nightly signal?

Integration Approach

Stack fit. Pure PHPUnit/Pest integration. Place the suite outside app/ (e.g. tests/conformance/suite/), add a dedicated <testsuite name="conformance"> entry in phpunit.xml, and tag every generated test with #[Group('conformance')] so it is excluded from the default fast run and from watch-mode. Nothing conflicts with Laravel's existing test bootstrap; the JSON tree must be excluded from autoload and from any file-watcher to avoid slowing local development.

Migration path. Phase 0 — spike: vendor a pinned snapshot, implement the retriever, run only draft2020-12 (or whichever draft matches production) non-optional cases, and record a baseline pass rate. Phase 1 — make that subset a CI gate scoped to PRs touching schema/validation code. Phase 2 — extend to optional/format and output-schema tests, or move the full matrix to a nightly job with a published pass-rate trend.

Compatibility. No install footprint, so PHP 8.1–8.4 and any Laravel version are fine. Two PHP-specific traps: json_decode precision on large integers (use JSON_BIGINT_AS_STRING where the case expects exact numeric comparison) and unicode escaping differences between mbstring/intl configurations. If format tests are included, intl and a format-assertion library become hard test-time requirements.

Sequencing. (1) Pin the commit SHA and document it. (2) Build the remote-ref retriever against remotes/. (3) Write the directory-walking data provider with memoized directory scans — do not re-stat thousands of files per test. (4) Author conformance-skip.php with a one-line reason per skipped case. (5) Wire the CI job with sharding by draft directory. (6) Add a small report artifact so pass rate per draft is visible over time.

Operational Impact

Maintenance. Upstream has not tagged a release since May 2023; treat the corpus as stable-by-freeze rather than actively maintained. The real maintenance surface is our runner and skip-list — budget 2–4 engineer-days initially and roughly one day per quarter for review and validator upgrades.

Support. No vendor support channel. Upstream issues are spec-level debates among implementers; expect to resolve ambiguity internally, not upstream. Ownership should sit with the platform/API team, not with whoever happened to write the runner.

Scaling. Thousands of cases across five drafts. In PHP, a naive run is minutes, not seconds. Shard by draft directory across parallel CI runners, cache the downloaded snapshot between jobs, and stream files per case rather than loading the whole tree into memory.

Failure modes. Skip-list creep silently erasing coverage; offline CI breaking remote-ref cases; an unpinned suite bump flipping expectations; format tests failing for environmental reasons (missing intl, ICU version) and being misread as validator bugs; float/large-integer precision producing false negatives; and the whole suite being mistaken for a validator by new contributors and wired into application runtime.

Ramp-up. Conceptually light for anyone fluent in JSON Schema — half a day to run the suite and read failures. The non-obvious skill is triage: deciding whether a red test is a validator defect, a test-suite ambiguity, or an intentionally unsupported keyword, and recording that judgment in the skip-list rather than deleting the test.