Airflow

software

★★★★½4.4(123 reviews)

About

Open-source workflow orchestration platform for scheduling and monitoring data pipelines. Define DAGs in Python for ML data workflows.

Overview

Apache Airflow is the industry-standard workflow orchestrator for data pipelines, offering unmatched extensibility and community support, though its complexity and resource requirements can be challenging for smaller teams.

orchestration data-pipelines workflow scheduling python

Pros

+Industry standard
+Python DAGs
+Massive plugin ecosystem
+Free

Cons

-Complex to operate
-DAG serialization issues
-Resource intensive

Visit website

This may be an affiliate link — the creator and GuruStacks may earn a commission, at no extra cost to you. Learn more

Details

Typesoftware

Pricing

Model

open source

Platforms

linuxmac

Community

Listed in Stacks

AI Model Training

Complete toolkit for training, fine-tuning, and deploying AI/ML models — from experiment tracking and GPU compute to data labeling, model serving, and vector databases.

by Gurustacks0

Data Scientist

Essential tools for data scientists covering notebooks, ML frameworks, data visualization, feature stores, experiment tracking, and more.

by Gurustacks0

Similar tools

View alternatives →

Prefect

4.5

orchestrationdata-pipelinesworkflow

Modern workflow orchestration framework that makes data pipelines easy to build, run, and observe. Pythonic API with hybrid execution.

Great Expectations

4.2

data-qualitytestingdata-governance

Data quality platform helping teams validate data across pipelines, catch issues early, and maintain governance with cloud and open-source solutions.

Dagster

4.2

orchestrationdata-pipelinesmodern

Modern data orchestration platform with software-defined assets, type checking, and built-in testing. A modern alternative to Airflow.

ComfyUI

4.4

aistable-diffusionimage-generation

Powerful and modular node-based visual editor for designing and executing advanced AI image and video generation pipelines using Stable Diffusion and other models.

CrewAI

4.4

multi-agentai-agentsopen-source

Multi-agent platform for enterprises to build, deploy, and monitor teams of autonomous AI agents collaborating on complex tasks, with both an open-source Python framework and a managed cloud platform.

AutoGen

4.4

open-sourcemulti-agentmicrosoft

Microsoft's open-source multi-agent framework that enables multiple LLM-powered agents to collaborate, debate, and self-correct on complex tasks through automated conversation and tool use.