Twitter Streaming Api Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require spatie/twitter-streaming-api
   

2. Authentication: Register an app on Twitter Developer Portal to obtain:

   • BEARER_TOKEN (Project > Keys and tokens)
   • API_KEY and API_SECRET_KEY (Project > Keys and tokens)

3. First Use Case: Stream public tweets containing a keyword (e.g., @spatie_be):

   use Spatie\TwitterStreamingApi\PublicStream;
   
   $stream = PublicStream::create(
       env('TWITTER_BEARER_TOKEN'),
       env('TWITTER_API_KEY'),
       env('TWITTER_API_SECRET_KEY')
   );
   
   $stream->whenHears('@spatie_be', function (array $tweet) {
       // Handle tweet (e.g., log, store in DB, or trigger a job)
       logger()->info('Mentioned!', ['tweet' => $tweet]);
   })->startListening();
   

4. Where to Look First:

   • README for basic examples.
   • API Docs for available stream types (PublicStream, UserStream, FilterStream).
   • Environment Variables for config.

Implementation Patterns

Core Workflows

1. Stream Types:

   • Public Stream: Listen to public tweets (e.g., keywords, hashtags).

     PublicStream::create(...)->whenHears('#laravel')->startListening();
     

   • User Stream: Listen to actions of a specific user (e.g., likes, follows).

     UserStream::create(...)->whenUserLikes(function (array $like) { ... })->startListening();
     

   • Filter Stream: Combine multiple rules (keywords, follow/ignore lists).

     FilterStream::create(...)->addRule('track', ['#php', '#laravel'])->startListening();
     

2. Event Handling:

   • Chain when* methods for specific actions (e.g., whenHears, whenUserFollows).
   • Use closures or class methods for callbacks:

     $stream->whenHears('keyword', [TweetHandler::class, 'handleMention']);
     

3. Long-Running Processes:

   • Run streams in a separate process (e.g., Laravel Queues, Supervisor) to avoid HTTP timeouts:

     // Dispatch a job to start the stream
     StartTwitterStreamJob::dispatch($bearerToken, $apiKey, $apiSecretKey);
     

   • Use Laravel Events to decouple stream logic:

     event(new TweetMentioned($tweet));
     

4. Data Persistence:

   • Store tweets in a database using Eloquent models or queued jobs:

     $stream->whenHears('keyword', function (array $tweet) {
         Tweet::create([
             'user_id' => $tweet['user']['id'],
             'text' => $tweet['text'],
             // ...
         ]);
     });
     

   • Batch inserts for performance:

     $tweets = [];
     $stream->whenHears('keyword', function (array $tweet) use (&$tweets) {
         $tweets[] = $tweet;
         if (count($tweets) >= 100) {
             Tweet::insert($tweets);
             $tweets = [];
         }
     });
     

5. Error Handling:

   • Wrap streams in try-catch blocks to handle disconnections:

     try {
         $stream->startListening();
     } catch (\Spatie\TwitterStreamingApi\Exceptions\ConnectionException $e) {
         $this->reconnectAfterDelay();
     }
     

   • Implement reconnection logic with exponential backoff.

6. Testing:

   • Mock the stream in unit tests using Mockery or Laravel's HTTP testing:

     $mockStream = Mockery::mock(\Spatie\TwitterStreamingApi\PublicStream::class);
     $mockStream->shouldReceive('startListening')->andReturnSelf();
     

Gotchas and Tips

Pitfalls

1. Rate Limits:

   • Twitter enforces rate limits (e.g., 500 connections/hour for streaming).
   • Solution: Implement retry logic with delays or use multiple streams with different credentials.

2. Connection Drops:

   • Streams may disconnect due to network issues or Twitter’s side.
   • Solution: Use Laravel’s queue:work --daemon or a process manager (e.g., Supervisor) to auto-restart failed jobs.

3. Authentication Changes:

   • Twitter’s API keys/tokens may expire or require updates.
   • Solution: Store credentials in .env and use Laravel’s env() helper. Monitor for deprecation notices.

4. Data Format:

   • Raw tweet data is nested and may change with Twitter API updates.
   • Solution: Use data_get() or flatten the array for consistency:

     $screenName = data_get($tweet, 'user.screen_name');
     

5. Memory Leaks:

   • Long-running streams can accumulate data in memory.
   • Solution: Process data in chunks or use database storage immediately.

6. Time Zone Handling:

   • Tweet timestamps are in UTC. Convert to local time if needed:

     $createdAt = Carbon::parse($tweet['created_at'])->timezone('America/New_York');
     

Debugging

1. Log Raw Data:

   • Log the full tweet payload to debug unexpected structures:

     logger()->debug('Raw tweet', ['tweet' => $tweet]);
     

2. Check HTTP Headers:

   • Verify the Authorization header is correctly set with the bearer token:

     $stream->setBearerToken(env('TWITTER_BEARER_TOKEN'));
     

3. Twitter API Status:

   • Monitor Twitter API status for outages.

Tips

1. Environment Variables:

   • Use Laravel’s .env for credentials:

     TWITTER_BEARER_TOKEN=your_bearer_token
     TWITTER_API_KEY=your_api_key
     TWITTER_API_SECRET_KEY=your_api_secret
     

2. Extending the Package:

   • Add custom stream rules by extending the base Stream class:

     class CustomStream extends \Spatie\TwitterStreamingApi\Stream {
         public function whenCustomEvent($callback) {
             $this->on('custom.event', $callback);
             return $this;
         }
     }
     

3. Performance:

   • Use database indexing for frequently queried tweet fields (e.g., user_id, created_at).
   • For high-volume streams, consider Elasticsearch or Redis for real-time analytics.

4. Security:

   • Restrict API keys to specific IPs or use Laravel’s API token for additional security.

5. Monitoring:

   • Track stream uptime and tweet volume with Laravel’s tasks or a monitoring tool (e.g., Datadog, Sentry).

6. Fallbacks:

   • Implement a fallback to a polling-based approach if streaming fails:

     if (!$stream->isConnected()) {
         $this->fallbackToPolling();
     }