Skip to content
Softwaregeek-SamPublic

About

Flowforge is a distributed job orchestration tool.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

12 Commits

Folders and files

Repository files navigation

FlowForge

A distributed job orchestration platform inspired by Apache Airflow, built from scratch in Java and Spring Boot.

FlowForge lets you define workflows as directed acyclic graphs (DAGs) of tasks, schedules them on cron expressions, and executes them across a pool of distributed workers — with coordination, idempotency, and failure isolation designed in from the start rather than added later.

Status: under active development. The DAG Registry module is complete. The Scheduler module is partially implemented. The Worker module has not been started. Nothing currently executes end to end. See Roadmap.

Table of contents

Why this exists

This is a solo-built project, not a company-backed system, and it isn't trying to compete with Airflow or Temporal. The point is to implement the parts of a distributed orchestrator that are actually hard to get right: coordinating multiple scheduler instances without double-scheduling a DAG, making task creation idempotent under retries, keeping message dispatch outside the database transaction so a rollback can't orphan a queue message, and building entities that enforce their own state machine instead of trusting every caller to set fields correctly.

Features

  • DAG definitions as YAML files, scanned and parsed at startup
  • Cycle detection (DFS) before a DAG is registered
  • Cron-based scheduling, coordinated across instances via PostgreSQL advisory locks
  • Dependency-aware task eligibility — a task becomes runnable only once every upstream dependency has succeeded
  • Idempotent task_run creation, safe against duplicate scheduler ticks
  • RabbitMQ-based task dispatch to a worker pool
  • Worker pool using the competing-consumers pattern
  • Heartbeat-based dead worker detection and task reassignment
  • DAG run completion evaluation (cascading a failed task to its downstream tasks and the run itself)
  • REST API for triggering and inspecting runs

Architecture

 DAG Registry                 Scheduler                      Worker (planned)
 ─────────────                ─────────                      ────────────────
 YAML DAG files       ──▶     Cron evaluation        ──▶     Pulls task_run IDs
 parsed, validated,           PostgreSQL advisory             from RabbitMQ
 persisted to Postgres        locks for coordination          Executes the task
                               Creates task_run rows           Reports status back
                                      │
                                      ▼
                               ┌─────────────┐
                               │  RabbitMQ   │
                               │ task queue  │
                               └─────────────┘
                                      │
                                      ▼
                               ┌─────────────┐
                               │ PostgreSQL  │
                               │ dag_run /   │
                               │ task_run    │
                               └─────────────┘

The scheduler never pushes to the queue until its database transaction has committed — if task creation fails and rolls back, nothing gets dispatched for a row that doesn't exist.

Tech stack

  • Language / framework: Java 21, Spring Boot
  • Database: PostgreSQL 16, via JPA/Hibernate (schema validation only — Flyway owns the schema) and NamedParameterJdbcTemplate for queries too complex for a derived query
  • Message broker: RabbitMQ (replacing an earlier Redis-based design — see Roadmap for the local-infra migration)
  • Migrations: Flyway
  • Build: Maven
  • Testing: JUnit 5, Mockito, ArchUnit (module boundary enforcement)

Getting started

Prerequisites

  • JDK 21
  • Maven
  • Docker and Docker Compose

Running locally

git clone https://github.com/<your-username>/flowforge.git
cd flowforge

# Start PostgreSQL (and RabbitMQ, once the local compose file is migrated — see Roadmap)
docker compose up -d

mvn spring-boot:run

With the Worker module not yet built, running this today gets you DAG registration, scheduling, and task_run creation — nothing consumes off the queue yet, so tasks won't actually execute.

Running tests

mvn test

Unit tests mock all collaborators and run without a Spring context. Integration tests (where present) use Testcontainers against a real Postgres instance.

Project structure

flowforge/
├── src/main/java/dev/sumit/flowforge/
│   ├── registry/               # DAG file scanning, YAML parsing, validation, persistence
│   └── scheduler/
│       ├── dagrun/             # DagRun entity, status enum, repository
│       ├── task/
│       │   ├── creator/        # TaskRun entity, TaskRunCreator
│       │   └── eligibility/    # EligibleTask, EligibleTaskRepository
│       ├── service/            # SchedulerService, DagStateManager
│       ├── job/                 # SchedulerJob — @Scheduled entry point
│       ├── lock/                 # DagLockManager — Postgres advisory locks
│       ├── cron/                 # CronEvaluator
│       └── shared/               # Cross-cutting domain exceptions
├── src/main/resources/db/migration/   # Flyway migrations
└── src/test/java/...                   # Mirrors main, one test class per production class

(registry package name above is indicative — adjust to match your actual layout if it differs.)

Key design decisions

A few choices worth calling out, because the reasoning isn't obvious from the code alone:

  • Dispatch happens after the transaction commits, never inside it. Dispatching inside the transaction risks pushing a queue message for a row that gets rolled back.
  • Coordination uses PostgreSQL advisory locks, not a separate distributed lock service. One less moving part, and the lock lives next to the data it's protecting.
  • Entities enforce their own state transitions. DagRun and TaskRun expose methods like markRunning() / markSucceeded(), not public setters — an invalid transition throws instead of silently corrupting state.
  • Timestamps are Instant, not LocalDateTime. The scheduler and its workers aren't guaranteed to run in the same timezone-configured environment.
  • Application-layer collaborators depend on a repository directly when there's no logic beyond loading rows. A wrapper service only gets introduced once real logic accumulates there — not ahead of time, on the assumption it might.

Roadmap

  • TaskDispatcher: RabbitMQ-based dispatch (currently unimplemented after the Redis → RabbitMQ decision)
  • Worker module: competing consumers pulling from RabbitMQ, with a heartbeat mechanism
  • Dead worker detection and task reassignment
  • DAG run completion evaluation — cascading a permanently failed task to its downstream tasks and the run itself
  • Migrate local Docker Compose from Redis to RabbitMQ
  • REST API for triggering and inspecting DAG runs
  • Full test coverage across the Scheduler module
  • HLD / LLD / ADR documentation

About

Flowforge is a distributed job orchestration tool.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages