Airflow Dag Run Bundle Laravel Package

Product Decisions This Supports

• Asynchronous Data Processing Pipeline: Enables triggering Airflow DAGs from Symfony applications to offload heavy computations (e.g., report generation, data exports) without blocking user requests.
• Decoupled Architecture: Supports a microservices or event-driven design by allowing Symfony to delegate workflows to Airflow while receiving async callbacks (e.g., export filenames) via Symfony’s Messenger component.
• Roadmap for Scalability: Justifies investing in Airflow for batch/ETL processes if the team lacks dedicated workflow orchestration tools, reducing reliance on custom cron jobs or synchronous APIs.
• Build vs. Buy: Favors "buy" for teams already using Symfony + Airflow, avoiding reinventing HTTP-based DAG triggers and callback handling.
• Use Cases:
  • User-triggered exports (e.g., "Generate my report" button → Airflow handles the heavy lifting).
  • Scheduled data pipelines where Symfony acts as the orchestrator for user-facing actions.
  • Hybrid systems where Symfony manages APIs and Airflow handles long-running tasks.

When to Consider This Package

• Adopt if:
  • Your Symfony app needs to trigger Airflow DAGs programmatically (e.g., via API endpoints or CLI commands).
  • You require asynchronous callbacks (e.g., notifying Symfony when a DAG completes with results like filenames).
  • Your team already uses Apache Airflow for workflows and wants to integrate it tightly with Symfony.
  • You’re using Symfony 6.4+ (Messenger component support) and need a pre-built solution over custom HTTP clients.
• Look elsewhere if:
  • You need real-time DAG monitoring (this bundle focuses on triggering, not UI/dashboards).
  • Your Airflow setup is headless or lacks HTTP API access (e.g., Kubernetes-based with no REST endpoint).
  • You require complex DAG parameterization (the bundle supports basic dag_ids; custom logic may need extensions).
  • You’re on Symfony < 6.4 (Messenger routing limitations apply; see README warning).
  • Your use case fits Airflow’s native REST API directly (e.g., no Symfony layer needed).

How to Pitch It (Stakeholders)

For Executives: "This bundle lets our Symfony apps delegate heavy, time-consuming tasks (like data exports or report generation) to Airflow—without blocking user requests. It’s like adding a ‘fire-and-forget’ button for complex workflows, with callbacks to notify our app when tasks complete. This reduces server load, improves responsiveness, and aligns with our existing Airflow investment. Low risk: it’s a thin wrapper around Airflow’s API, with minimal maintenance overhead."

For Engineering: *"This solves the problem of triggering Airflow DAGs from Symfony cleanly, with built-in support for async result handling via Messenger. Key benefits:

• No custom HTTP client code: Handles auth, retries, and basic DAG execution.
• Decoupled workflows: Symfony can kick off long-running tasks and receive updates via messages (e.g., RabbitMQ, Doctrine).
• Config-driven: Easy to add/remove DAGs via YAML.
• Symfony-native: Integrates with Flex, Messenger, and config systems we already use. Tradeoff: Limited to basic DAG triggers; for advanced use cases, we’d need to extend it. But it’s a great starting point for async pipelines."*