A distributed job orchestration platform inspired by Apache Airflow, built from scratch in Java and Spring Boot.
FlowForge lets you define workflows as directed acyclic graphs (DAGs) of tasks, schedules them on cron expressions, and executes them across a pool of distributed workers — with coordination, idempotency, and failure isolation designed in from the start rather than added later.
Status: under active development. The DAG Registry module is complete. The Scheduler module is partially implemented. The Worker module has not been started. Nothing currently executes end to end. See Roadmap.
- Why this exists
- Features
- Architecture
- Tech stack
- Getting started
- Project structure
- Key design decisions
- Roadmap
- Contributing
- License
This is a solo-built project, not a company-backed system, and it isn't trying to compete with Airflow or Temporal. The point is to implement the parts of a distributed orchestrator that are actually hard to get right: coordinating multiple scheduler instances without double-scheduling a DAG, making task creation idempotent under retries, keeping message dispatch outside the database transaction so a rollback can't orphan a queue message, and building entities that enforce their own state machine instead of trusting every caller to set fields correctly.
- DAG definitions as YAML files, scanned and parsed at startup
- Cycle detection (DFS) before a DAG is registered
- Cron-based scheduling, coordinated across instances via PostgreSQL advisory locks
- Dependency-aware task eligibility — a task becomes runnable only once every upstream dependency has succeeded
- Idempotent
task_runcreation, safe against duplicate scheduler ticks - RabbitMQ-based task dispatch to a worker pool
- Worker pool using the competing-consumers pattern
- Heartbeat-based dead worker detection and task reassignment
- DAG run completion evaluation (cascading a failed task to its downstream tasks and the run itself)
- REST API for triggering and inspecting runs
DAG Registry Scheduler Worker (planned)
───────────── ───────── ────────────────
YAML DAG files ──▶ Cron evaluation ──▶ Pulls task_run IDs
parsed, validated, PostgreSQL advisory from RabbitMQ
persisted to Postgres locks for coordination Executes the task
Creates task_run rows Reports status back
│
▼
┌─────────────┐
│ RabbitMQ │
│ task queue │
└─────────────┘
│
▼
┌─────────────┐
│ PostgreSQL │
│ dag_run / │
│ task_run │
└─────────────┘
The scheduler never pushes to the queue until its database transaction has committed — if task creation fails and rolls back, nothing gets dispatched for a row that doesn't exist.
- Language / framework: Java 21, Spring Boot
- Database: PostgreSQL 16, via JPA/Hibernate (schema validation only — Flyway owns the schema) and
NamedParameterJdbcTemplatefor queries too complex for a derived query - Message broker: RabbitMQ (replacing an earlier Redis-based design — see Roadmap for the local-infra migration)
- Migrations: Flyway
- Build: Maven
- Testing: JUnit 5, Mockito, ArchUnit (module boundary enforcement)
- JDK 21
- Maven
- Docker and Docker Compose
git clone https://github.com/<your-username>/flowforge.git
cd flowforge
# Start PostgreSQL (and RabbitMQ, once the local compose file is migrated — see Roadmap)
docker compose up -d
mvn spring-boot:runWith the Worker module not yet built, running this today gets you DAG registration, scheduling, and task_run creation — nothing consumes off the queue yet, so tasks won't actually execute.
mvn testUnit tests mock all collaborators and run without a Spring context. Integration tests (where present) use Testcontainers against a real Postgres instance.
flowforge/
├── src/main/java/dev/sumit/flowforge/
│ ├── registry/ # DAG file scanning, YAML parsing, validation, persistence
│ └── scheduler/
│ ├── dagrun/ # DagRun entity, status enum, repository
│ ├── task/
│ │ ├── creator/ # TaskRun entity, TaskRunCreator
│ │ └── eligibility/ # EligibleTask, EligibleTaskRepository
│ ├── service/ # SchedulerService, DagStateManager
│ ├── job/ # SchedulerJob — @Scheduled entry point
│ ├── lock/ # DagLockManager — Postgres advisory locks
│ ├── cron/ # CronEvaluator
│ └── shared/ # Cross-cutting domain exceptions
├── src/main/resources/db/migration/ # Flyway migrations
└── src/test/java/... # Mirrors main, one test class per production class
(registry package name above is indicative — adjust to match your actual layout if it differs.)
A few choices worth calling out, because the reasoning isn't obvious from the code alone:
- Dispatch happens after the transaction commits, never inside it. Dispatching inside the transaction risks pushing a queue message for a row that gets rolled back.
- Coordination uses PostgreSQL advisory locks, not a separate distributed lock service. One less moving part, and the lock lives next to the data it's protecting.
- Entities enforce their own state transitions.
DagRunandTaskRunexpose methods likemarkRunning()/markSucceeded(), not public setters — an invalid transition throws instead of silently corrupting state. - Timestamps are
Instant, notLocalDateTime. The scheduler and its workers aren't guaranteed to run in the same timezone-configured environment. - Application-layer collaborators depend on a repository directly when there's no logic beyond loading rows. A wrapper service only gets introduced once real logic accumulates there — not ahead of time, on the assumption it might.
-
TaskDispatcher: RabbitMQ-based dispatch (currently unimplemented after the Redis → RabbitMQ decision) - Worker module: competing consumers pulling from RabbitMQ, with a heartbeat mechanism
- Dead worker detection and task reassignment
- DAG run completion evaluation — cascading a permanently failed task to its downstream tasks and the run itself
- Migrate local Docker Compose from Redis to RabbitMQ
- REST API for triggering and inspecting DAG runs
- Full test coverage across the Scheduler module
- HLD / LLD / ADR documentation