Airflow Dag Run Bundle Laravel Package

Technical Evaluation

Architecture Fit

• Use Case Alignment: The bundle provides a Symfony-compatible HTTP client to trigger Airflow DAGs asynchronously, making it ideal for event-driven workflows where Symfony applications need to delegate long-running tasks (e.g., data exports, batch processing) to Airflow.
• Decoupling: Leverages Symfony Messenger for async execution, reducing coupling between Symfony and Airflow. The bundle dispatches DagRunMessageExecuted events, enabling downstream processing (e.g., notifications, file handling).
• Extensibility: Supports custom DAG parameters via configuration, allowing flexibility in triggering specific workflows. The airflow_dag_ids mapping (e.g., name:dag-id) enables granular control over DAG selection.
• Symfony Ecosystem Fit: Integrates natively with Symfony’s dependency injection, configuration system, and Messenger component, reducing boilerplate for PHP developers.

Integration Feasibility

• HTTP-Based Communication: Relies on Airflow’s REST API (or UI auth), which may require:
  • Airflow API Access: Ensure Airflow is configured with AIRFLOW__WEBSERVER__REST_API_ENABLED=True (default in modern versions).
  • Authentication: Supports basic auth (username/password) or could be extended for token/OAuth.
• Messenger Dependency: Requires Symfony Messenger (v6.4+ for full async support). For older versions, the bundle falls back to synchronous execution, which may limit scalability.
• Error Handling: Limited visibility into Airflow task failures unless paired with Airflow’s callback system or custom event listeners.

Technical Risk

• Airflow Version Compatibility: No explicit version constraints in the README. Risk of breaking changes if Airflow’s REST API evolves (e.g., deprecated endpoints, auth shifts).
• State Management: Bundle assumes Airflow DAGs are idempotent. Retries or duplicate triggers could cause unintended side effects (e.g., duplicate exports).
• Performance Overhead: HTTP calls to Airflow may introduce latency. Caching or local DAG state tracking could mitigate this.
• Security:
  • Hardcoded credentials in config/packages/ (unless using environment variables or Symfony’s parameter_bag).
  • No TLS validation by default (risk if airflow_host uses http://).
• Monitoring: Lack of built-in metrics/logging for DAG execution status. Requires custom instrumentation (e.g., logging DagRunMessageExecuted events).

Key Questions

1. Airflow Setup:
   • Is Airflow’s REST API enabled and accessible from Symfony?
   • What authentication method is preferred (basic auth, tokens, etc.)?
2. Async Requirements:
   • Does the team need guaranteed delivery of DAG triggers (e.g., with retries)?
   • How will failures be surfaced (e.g., dead-letter queues, alerts)?
3. Configuration Management:
   • How will airflow_password be secured (env vars, Vault, etc.)?
   • Are there plans to support dynamic DAG parameters (e.g., runtime overrides)?
4. Scaling:
   • Will multiple Symfony instances trigger the same DAGs? (Risk of duplicates.)
   • Is Messenger’s transport (e.g., RabbitMQ, Doctrine) scalable for expected load?
5. Observability:
   • How will DAG execution status be tracked (e.g., UI, logs, external monitoring)?
6. Alternatives:
   • Could Airflow’s KubernetesPodOperator or CeleryExecutor reduce HTTP overhead?
   • Is there a need for webhook-based triggers instead of polling?

Integration Approach

Stack Fit

• Symfony Core: Native integration with Symfony 6.4+ (Messenger, DI, Config). For older versions, synchronous fallback may suffice for low-volume use.
• Airflow: Compatible with Airflow 2.x+ (REST API). Tested with basic auth; extension needed for advanced auth (e.g., JWT).
• Messenger Transports: Supports any Symfony Messenger transport (e.g., async, doctrine, amqp). Choose based on reliability needs:
  • Async: For high throughput (e.g., RabbitMQ).
  • Doctrine: For simplicity (persistent retries).
• Database: No direct DB requirements, but Messenger transport may need a DB (e.g., Doctrine).

Migration Path

1. Assessment Phase:
   • Verify Airflow REST API is enabled (AIRFLOW__WEBSERVER__REST_API_ENABLED).
   • Test basic auth with curl or Postman to confirm DAG triggers work.
2. Symfony Setup:
   • Install bundle: composer require bluspark/airflow-dag-run-bundle.
   • Enable bundle in config/bundles.php.
   • Configure bluspark_airflow_dag_run.yaml with secure credentials (use %env(AIRFLOW_PASSWORD)%).
3. Messenger Configuration:
   • Add transport to messenger.yaml (e.g., async for simplicity).
   • Route DagRunMessageExecuted to the transport.
4. Service Integration:
   • Inject Bluspark\AirflowDagRunBundle\Scheduler\DagRunScheduler into services to trigger DAGs:

     $scheduler->triggerDag('export_dag', ['param' => 'value']);
     

5. Testing:
   • Mock Airflow responses in unit tests (e.g., using Guzzle middleware).
   • Test failure scenarios (e.g., Airflow downtime, auth failures).

Compatibility

• Symfony Versions:
  • 6.4+: Full async support.
  • <6.4: Synchronous execution (risk of timeouts for slow DAGs).
• Airflow Versions:
  • Tested with 2.x; validate against your Airflow version’s REST API docs.
• PHP: Requires PHP 8.1+ (Symfony 6.x baseline).

Sequencing

1. Phase 1: Basic Triggering
   • Implement core DAG triggers via DagRunScheduler.
   • Log execution results (e.g., DagRunMessageExecuted).
2. Phase 2: Async Processing
   • Configure Messenger transport for reliability.
   • Add retry logic for failed triggers (custom middleware).
3. Phase 3: Observability
   • Extend bundle to log DAG run IDs for tracking.
   • Integrate with APM (e.g., track DagRunMessageExecuted in Sentry).
4. Phase 4: Advanced Features
   • Dynamic DAG parameters from Symfony services.
   • Webhook callbacks for DAG completion (e.g., notify Symfony of export filenames).

Operational Impact

Maintenance

• Configuration Drift: Credentials and DAG IDs are YAML-based; use Symfony’s parameter_bag or env vars to centralize secrets.
• Dependency Updates:
  • Bundle is actively maintained (last release: 2025-06-26), but monitor for breaking changes.
  • Airflow API changes may require bundle updates.
• Logging:
  • Limited built-in logging; extend DagRunScheduler to log trigger payloads and responses.
  • Example:

    $logger->info('Triggered DAG', ['dag_id' => $dagId, 'params' => $params]);
    

Support

• Troubleshooting:
  • Auth Failures: Verify airflow_username/password and Airflow’s AIRFLOW__WEBSERVER__AUTH_BACKEND.
  • HTTP Errors: Check Airflow’s REST API logs (airflow scheduler/webserver logs).
  • Messenger Issues: Monitor transport queues for stuck messages.
• Documentation: README is minimal; create internal docs for:
  • DAG trigger workflows.
  • Error handling procedures.
• Vendor Lock-in: Bundle is lightweight; replacing it would require rewriting HTTP clients and Messenger logic.

Scaling

• Horizontal Scaling:
  • Symfony instances can safely trigger DAGs in parallel, but:
    • Duplicate Risk: Use Airflow’s run_id or external locks (e.g., Redis) to prevent duplicates.
    • Throttling: Airflow’s REST API may rate-limit requests; implement exponential backoff.
• Messenger Scaling:
  • Async transport (e.g., RabbitMQ) scales better than synchronous.
  • Monitor queue depth under load.
• Airflow Load:
  • High DAG trigger volume may stress Airflow’s scheduler. Consider:
    • DAG Tagging: Route triggers to specific Airflow workers.
    • Batch Processing: Group triggers if possible.

Failure Modes