Ecs Logging Laravel Package

Technical Evaluation

Architecture Fit

• Use Case Alignment: The elastic/ecs-logging package provides a standardized way to structure logs using the Elastic Common Schema (ECS), which is ideal for applications requiring observability, centralized logging, and analytics (e.g., Laravel apps integrated with ELK Stack, Datadog, or other SIEM tools).
• Laravel Synergy: Laravel’s logging system is extensible via monolog, and this package can be integrated as a log processor to transform raw logs into ECS-compliant JSON. This aligns well with Laravel’s structured logging needs (e.g., for debugging, monitoring, or compliance).
• Key Benefits:
  • Standardized Log Fields: Ensures consistency across microservices or multi-language stacks.
  • ELK/Datadog Compatibility: Reduces parsing overhead in log aggregation pipelines.
  • Context Enrichment: Supports adding metadata (e.g., request IDs, user roles) via Laravel’s Log::withContext().

Integration Feasibility

• Monolog Integration: The package is designed for monolog, Laravel’s default logger, making adoption straightforward. Can be plugged into Laravel’s logging stack via:

  $logger = new Monolog\Logger('name');
  $logger->pushProcessor(new \Elastic\EcsLog\Processor());
  

• Middleware/Event Hooks: Can be extended via Laravel’s logging middleware (e.g., LogRequest) or events (Logging events) to inject ECS fields dynamically.
• Database Logs: If using Laravel’s database log driver, ECS formatting would require a custom handler (higher effort but feasible).

Technical Risk

• Schema Rigidity: ECS is opinionated; over-structuring logs may require schema adjustments for Laravel’s dynamic data (e.g., nested relationships).
• Performance Overhead: JSON serialization adds minor latency (~1–5ms per log entry). Benchmark in production-like conditions.
• Dependency Bloat: Adds elastic/ecs-logging + monolog as dependencies. Assess if existing logging setup can absorb this without conflicts.
• Backward Compatibility: Existing log consumers (e.g., third-party parsers) may need updates to handle ECS fields.

Key Questions

1. Log Volume: Is the app high-throughput? ECS may impact performance at scale.
2. Existing Pipeline: Are logs already sent to ELK/Datadog? If not, is this a precursor to migration?
3. Custom Fields: Does Laravel’s data model map cleanly to ECS fields (e.g., error.stack_trace vs. Laravel’s exception formatting)?
4. Retention Policy: Will ECS-structured logs require changes to log storage (e.g., Elasticsearch index templates)?
5. Team Skills: Is the team familiar with ECS/ELK? Training may be needed for debugging.

Integration Approach

Stack Fit

• Primary Fit: Laravel + Monolog + ELK Stack/Datadog/Splunk.
• Secondary Fit: Any PHP app using monolog (e.g., Symfony, Lumen).
• Non-Fit: Apps using non-monolog loggers (e.g., error_log(), custom file handlers) or databases as primary log sinks.

Migration Path

1. Phase 1: Pilot

   • Integrate ECS processor in a non-critical environment (e.g., staging).
   • Test with a subset of loggers (e.g., single driver for debugging).
   • Validate ECS output matches expectations (e.g., event.dataset for Laravel-specific logs).

2. Phase 2: Full Rollout

   • Replace default log handlers with ECS-processed ones in config/logging.php:

     'channels' => [
         'ecs' => [
             'driver' => 'single',
             'handler' => Monolog\Handler\StreamHandler::class,
             'processors' => [
                 new \Elastic\EcsLog\Processor(),
             ],
         ],
     ],
     

   • Update log middleware/events to include ECS context (e.g., ecs.tags.environment).

3. Phase 3: Pipeline Sync

   • Configure ELK/Datadog to ingest ECS logs (e.g., Elasticsearch ILM policies for retention).
   • Deprecate legacy log formats in dashboards/alerts.

Compatibility

• Laravel Versions: Compatible with Laravel 8+ (monolog v2+). Test with Laravel 9/10 for breaking changes.
• PHP Versions: Requires PHP 7.4+. No known conflicts with Laravel’s PHP 8.0+ features.
• Existing Processors: Can coexist with other monolog processors (e.g., UuidProcessor) but order matters (ECS should run last).

Sequencing

1. Dependency Installation:

   composer require elastic/ecs-logging monolog/monolog
   

2. Configuration:
   • Update config/logging.php to include ECS processor.
   • Add ECS context to critical loggers (e.g., Log::withContext(['ecs' => ['service.name' => 'laravel-app']])).
3. Validation:
   • Check logs in Elasticsearch/Kibana for correct ECS field population.
   • Verify no critical fields are missing (e.g., error.message, http.request.method).
4. Monitoring:
   • Set up alerts for malformed ECS logs (e.g., log.level missing).

Operational Impact

Maintenance

• Pros:
  • Reduced Parsing: ECS logs are self-describing, cutting down on logstash/grok overhead.
  • Centralized Schema: ECS fields can be version-controlled (e.g., via ecs.version).
• Cons:
  • Schema Drift Risk: Adding custom ECS fields may require coordination across services.
  • Dependency Updates: Elastic’s ECS package may evolve; monitor for breaking changes.

Support

• Debugging:
  • ECS logs enable richer queries (e.g., log.level: error AND service.name: api). However, debugging may require familiarity with ECS field names.
  • Use Log::debug() with ECS context to trace issues in production.
• Tooling:
  • ELK Stack’s Discover or Dev Tools Console becomes primary for log analysis.
  • Third-party tools (e.g., Humio, Splunk) may need ECS-specific configurations.

Scaling

• Performance:
  • ECS adds ~1–5ms per log entry. For high-volume apps (>10K logs/sec), consider:
    • Async Processing: Use Monolog\Handler\AsyncHandler to batch logs.
    • Sampling: Log only critical events at scale (e.g., error level only).
• Storage:
  • ECS logs are larger than plaintext. Optimize Elasticsearch indices with:
    • Field Mappings: Use keyword for high-cardinality fields (e.g., user.id).
    • Retention: Set ILM policies to age off old logs.

Failure Modes

Ramp-Up

• Developer Onboarding:
  • Document ECS field usage (e.g., ecs.tags.environment = 'production').
  • Provide examples for common Laravel scenarios (e.g., logging API requests, exceptions).
• Operational Training:
  • Train SREs on querying ECS logs in Kibana (e.g., event.dataset: laravel AND response.status_code: 500).
  • Share runbooks for common issues (e.g., "Logs not appearing in Elasticsearch").
• Phased Adoption:
  • Start with non-production logs (e.g., debug level) before enforcing ECS for all logs.