Extractor Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require helgesverre/extractor
   

   Publish the config file (if customization is needed):

   php artisan vendor:publish --provider="HelgeSverre\Extractor\ExtractorServiceProvider"
   

2. First Use Case: Extract text from an image (OCR) and parse structured data:

   use HelgeSverre\Extractor\Facades\Extractor;
   use HelgeSverre\Extractor\Facades\Text;
   
   // OCR from an image
   $imagePath = storage_path('app/restaurant_menu.png');
   $extractedText = Text::textract($imagePath);
   
   // Extract structured data (e.g., menu items)
   $menuData = Extractor::extract($extractedText, [
       'fields' => [
           'items' => [
               'type' => 'array',
               'items' => [
                   'name' => 'string',
                   'price' => 'float',
                   'description' => 'string?',
               ],
           ],
       ],
   ]);
   

3. Key Files to Review:

   • config/extractor.php: Configure OpenAI API keys, default models, and Textract settings.
   • app/Extractors/: Custom extractors (if extending functionality).

Implementation Patterns

Core Workflows

1. Text Extraction from Files

Use Text::extract() for plain text, PDFs, or documents:

$pdfText = Text::extract(storage_path('app/invoice.pdf'));

2. Structured Data Extraction

Define a schema and let the package handle parsing:

$schema = [
    'fields' => [
        'invoice_number' => 'string',
        'total_amount' => 'float',
        'items' => [
            'type' => 'array',
            'items' => [
                'product' => 'string',
                'quantity' => 'int',
                'unit_price' => 'float',
            ],
        ],
    ],
];

$data = Extractor::extract($pdfText, $schema);

3. OCR for Images

Leverage AWS Textract (configured in config/extractor.php):

$imageText = Text::textract(storage_path('app/receipt.jpg'));

4. Web Content Extraction

Fetch and parse HTML content:

$webText = Text::fromUrl('https://example.com/article');
$structuredData = Extractor::extract($webText, $schema);

5. Spatie Data Integration

Return structured data as a Spatie\Data\Data object:

$data = Extractor::extractAsData($pdfText, $schema);
// Usage: $data->invoice_number, $data->items[0]->product

Integration Tips

API-Driven Workflows

Chain extraction with API calls (e.g., store results in a database):

$extracted = Extractor::extract($text, $schema);
Invoice::create($extracted);

Batch Processing

Process multiple files in a loop:

foreach (Storage::files('invoices') as $file) {
    $text = Text::extract($file);
    $data = Extractor::extract($text, $schema);
    // Process $data...
}

Custom Extractors

Extend the base Extractor class for domain-specific logic:

namespace App\Extractors;

use HelgeSverre\Extractor\Extractor as BaseExtractor;

class InvoiceExtractor extends BaseExtractor {
    protected function customizePrompt($text, $schema) {
        return "Extract invoice details from: {$text}. Focus on tax lines.";
    }
}

Register in config/extractor.php:

'extractors' => [
    'invoice' => App\Extractors\InvoiceExtractor::class,
],

Usage:

$invoice = Extractor::extractWith('invoice', $text, $schema);

Error Handling

Wrap calls in try-catch for robustness:

try {
    $data = Extractor::extract($text, $schema);
} catch (\HelgeSverre\Extractor\Exceptions\ExtractionFailed $e) {
    Log::error("Extraction failed: {$e->getMessage()}");
    // Fallback logic (e.g., manual review)
}

Gotchas and Tips

Pitfalls

1. API Rate Limits:

   • OpenAI/Textract have strict rate limits. Monitor usage in config/extractor.php:

     'openai' => [
         'max_tokens' => 4000, // Adjust based on your model
         'temperature' => 0.3, // Lower for deterministic outputs
     ],
     

   • Implement retries with exponential backoff for failed requests.

2. Schema Design:

   • Overly complex schemas may reduce accuracy. Start simple and iterate.
   • Use optional fields (string?) for non-critical data to avoid extraction failures.

3. File Size Limits:

   • Large PDFs (>10MB) may fail. Pre-process or split files if needed.

4. OCR Limitations:

   • Textract may struggle with low-quality images. Pre-process images (e.g., using imagemagick) for better results.

5. Cost Management:

   • OpenAI charges per token. Log token usage to track costs:

     Extractor::extract($text, $schema, function ($response) {
         Log::info("Tokens used: {$response->usage->total_tokens}");
     });
     

Debugging Tips

1. Inspect Raw Responses: Enable debug mode in config/extractor.php:

   'debug' => env('EXTRACTOR_DEBUG', false),
   

   Logs raw OpenAI/Textract responses to storage/logs/extractor.log.

2. Validate Schemas: Use JSON Schema validators to test schemas before extraction:

   composer require webonyx/graphql-php
   

   Example validation:

   use Webonyx\GraphQLPHP\Validator\Validator;
   
   $validator = new Validator();
   $isValid = $validator->validate($schema);
   

3. Fallback for Failures: Implement a fallback to manual review or simpler extraction:

   $data = Extractor::extract($text, $schema);
   if ($data->isIncomplete()) {
       return redirect()->route('manual-review', ['text' => $text]);
   }
   

Extension Points

1. Custom Prompt Engineering: Override the default prompt template in your extractor:

   protected function getPrompt($text, $schema) {
       return "Extract data from: {$text}. Follow this schema strictly: {$schema}. Return only JSON.";
   }
   

2. Post-Processing: Use the afterExtract hook to transform data:

   Extractor::extract($text, $schema, function ($data) {
       $data->total = array_sum(array_column($data->items, 'unit_price'));
       return $data;
   });
   

3. Model-Specific Config: Dynamically switch OpenAI models based on input size:

   Extractor::extract($text, $schema, null, [
       'model' => strlen($text) > 8000 ? 'gpt-4' : 'gpt-3.5-turbo',
   ]);
   

4. Caching: Cache extraction results for static content (e.g., product descriptions):

   $cacheKey = "extracted_{$schemaHash}_{$textHash}";
   $data = Cache::remember($cacheKey, now()->addHours(1), function () use ($text, $schema) {
       return Extractor::extract($text, $schema);
   });
   

5. Testing: Mock the OpenAI client for unit tests:

   use HelgeSverre\Extractor\OpenAIClient;
   
   $mockClient = Mockery::mock(OpenAIClient::class);
   $mockClient->shouldReceive('chat')
       ->andReturn(['choices' => [['message' => ['content' => '{"key": "value"}']]]]);
   
   $this->app->instance(OpenAIClient::class, $mockClient);