Laravel Url Ai Transformer Laravel Package

Getting Started

Minimal Setup

1. Installation: Add the package via Composer:

   composer require spatie/laravel-url-ai-transformer
   

   Publish the config:

   php artisan vendor:publish --provider="Spatie\LaravelUrlAiTransformer\UrlAiTransformerServiceProvider"
   

2. Configure AI Provider: Set your AI service (e.g., OpenAI, Anthropic) in .env:

   URL_AI_TRANSFORMER_PROVIDER=openai
   URL_AI_TRANSFORMER_MODEL=gpt-4
   

3. First Use Case: Transform a URL into structured data:

   use Spatie\LaravelUrlAiTransformer\Support\Transform;
   use Spatie\LaravelUrlAiTransformer\Transformers\LdJsonTransformer;
   
   Transform::urls('https://example.com/blog/post')
       ->usingTransformers(new LdJsonTransformer);
   

4. Run the Command:

   php artisan transform-urls
   

5. Retrieve Results:

   $result = \Spatie\LaravelUrlAiTransformer\Models\TransformationResult::forUrl(
       'https://example.com/blog/post',
       'ldJson'
   );
   

Where to Look First

• Config: config/url-ai-transformer.php for AI provider settings.
• Transformers: Built-in transformers (LdJsonTransformer, ImageTransformer) in vendor/spatie/laravel-url-ai-transformer/src/Transformers.
• Database: transformation_results table for stored results.

Implementation Patterns

Core Workflow

1. Register URLs:

   Transform::urls('https://example.com/page1', 'https://example.com/page2')
       ->usingTransformers(new LdJsonTransformer);
   

   • Use closures for dynamic URLs:

     Transform::urls(fn() => Article::published()->pluck('url')->toArray())
         ->usingTransformers(new LdJsonTransformer);
     

2. Run Transformations:

   • Queued (default): php artisan transform-urls (dispatches jobs).
   • Synchronous (debugging):

     php artisan transform-urls --now
     

3. Retrieve Results:

   $result = TransformationResult::forUrl('https://example.com', 'ldJson');
   

Integration Tips

• Scheduled Updates: Add to app/Console/Kernel.php:

  protected function schedule(Schedule $schedule)
  {
      $schedule->command('transform-urls')->dailyAt('02:00');
  }
  

• Conditional Transformations: Override shouldRun in custom transformers:

  public function shouldRun(): bool
  {
      return $this->urlContent->contains('important-keyword');
  }
  

• Batch Processing: Use Laravel queues with batch jobs:

  Transform::urls($urls)->batch(50)->usingTransformers(new LdJsonTransformer);
  

• API Endpoints: Create routes to fetch results:

  Route::get('/transformations/{url}', function ($url) {
      return TransformationResult::forUrl($url, 'ldJson')->result;
  });
  

Custom Transformer Example

// app/Transformers/CustomSummaryTransformer.php
namespace App\Transformers;

use Spatie\LaravelUrlAiTransformer\Transformers\Transformer;
use Spatie\LaravelUrlAiTransformer\Support\Config;

class CustomSummaryTransformer extends Transformer
{
    public function transform(): void
    {
        $response = Prism::text()
            ->using(Config::aiProvider(), Config::aiModel())
            ->withPrompt($this->getPrompt())
            ->asText();

        $this->transformationResult->result = $response->text;
    }

    public function getPrompt(): string
    {
        return "Summarize this content in 5 bullet points:\n\n" . $this->urlContent;
    }

    public function type(): string
    {
        return 'custom_summary';
    }
  }

Gotchas and Tips

Pitfalls

1. Rate Limits:

   • AI providers (e.g., OpenAI) have rate limits. Use exponential backoff in custom jobs:

     public $backoff = [60, 120, 300];
     

2. URL Fetching Failures:

   • Ensure URLs are accessible (CORS, robots.txt, or server restrictions may block scraping).
   • Handle exceptions in custom transformers:

     try {
         $this->transform();
     } catch (\Exception $e) {
         $this->transformationResult->markAsFailed($e);
     }
     

3. Token Limits:

   • Long URLs/content may exceed AI token limits. Pre-process content:

     $this->urlContent = Str::limit($this->urlContent, 4000);
     

4. Transformer Type Conflicts:

   • Avoid naming collisions (e.g., MyTransformer vs. MyTransformerV2). Use explicit type():

     public function type(): string { return 'my_transformer_v2'; }
     

Debugging Tips

1. Check Logs:

   • Enable Laravel debug mode and check storage/logs/laravel.log for job failures.

2. Inspect Database:

   • Query transformation_results for failed transformations:

     SELECT * FROM transformation_results
     WHERE latest_exception_seen_at IS NOT NULL;
     

3. Test Locally:

   • Use --now flag to debug synchronously:

     php artisan transform-urls --now --url="https://example.com"
     

4. Mock AI Responses:

   • Override Prism in tests:

     Prism::shouldReceive('text')->andReturn(new MockResponse());
     

Extension Points

1. Custom Job Classes:

   • Extend ProcessTransformerJob to add middleware or retry logic:

     // app/Jobs/CustomProcessTransformerJob.php
     use Spatie\LaravelUrlAiTransformer\Jobs\ProcessTransformerJob;
     
     class CustomProcessTransformerJob extends ProcessTransformerJob
     {
         public function middleware(): array
         {
             return [new \Illuminate\Bus\Middleware\ThrottleJobs(5)];
         }
     }
     

   • Register in config:

     'process_transformer_job' => App\Jobs\CustomProcessTransformerJob::class,
     

2. Pre/Post-Processing:

   • Override beforeTransform/afterTransform in custom transformers:

     public function beforeTransform(): void
     {
         $this->urlContent = $this->cleanContent($this->urlContent);
     }
     

3. Event Listeners:

   • Listen for transformation events:

     // app/Providers/EventServiceProvider.php
     protected $listen = [
         \Spatie\LaravelUrlAiTransformer\Events\TransformationCompleted::class => [
             \App\Listeners\LogTransformation::class,
         ],
     ];
     

4. Filtering:

   • Use command flags to limit transformations:

     # Transform only URLs matching a pattern
     php artisan transform-urls --url="*/blog/*"
     
     # Transform only specific transformer types
     php artisan transform-urls --transformer="ldJson"
     

Configuration Quirks

1. AI Provider Switching:

   • Ensure your .env matches the config:

     URL_AI_TRANSFORMER_PROVIDER=openai
     URL_AI_TRANSFORMER_MODEL=gpt-4
     

   • For custom providers, implement Spatie\LaravelUrlAiTransformer\Contracts\AiProvider.

2. Queue Connections:

   • Set queue_connection in config to avoid default queue delays:

     'queue_connection' => 'redis',
     

3. Storage Paths:

   • For ImageTransformer, ensure storage/app/public/transformed-images is writable.

Performance Tips

1. Batch Processing:

   • Use --batch flag to limit concurrent jobs:

     php artisan transform-urls --batch=20
     

2. Cache Results:

   • Cache TransformationResult queries:

     $result = Cache::remember("transform_{$url}_{$type}", now()->addHours(1), function() use ($url, $type) {
         return TransformationResult::forUrl($url, $type);
     });
     

3. Optimize Prompts:

   • Keep prompts concise to avoid token limits. Example:

     public function getPrompt(): string
     {
         return "Extract key points from this text (max 3 sentences):\n\n" . $this->urlContent;
     }