BuildFlow

BuildFlow is an open-source Python framework designed to simplify building scalable data pipelines. It abstracts away complex infrastructure and cloud resource management, allowing developers to define data processing workflows by decorating Python functions. Inspired by experiences with Apache Beam, BuildFlow focuses on common large-scale use cases with a lightweight API and provides generic and use case-driven I/O connectors, primarily supporting Google Cloud services initially. It uses Ray for distributed processing, achieving approximately 5,000 messages per second on a 4-core VM, and supports both stateless and stateful processing via a Processor class API. BuildFlow offers a managed cloud service (in private beta) and plans a VS Code extension to enable visual pipeline building and developer tooling. It targets data scientists and engineering teams seeking an easy, scalable alternative to complex streaming frameworks.

platform:web platform:linux platform:macos platform:windows platform:python form:library form:cli-tool form:api form:saas feature:stream-processing feature:data-pipelines feature:scalability feature:cloud-integration feature:open-source feature:stateful-processing feature:stateless-processing feature:resource-management feature:decorator-based-api integration:google-cloud integration:gcp integration:bigquery integration:pubsub target:data-scientists target:developers target:engineering-teams pricing:free pricing:usage-based

Features

Stream Processing

Data Pipelines

Scalability

Cloud Integration

Open Source

Stateful Processing

Stateless Processing

Resource Management

Decorator Based API

Testimonies

No testimonies available for this tool yet.

Basic Info

Category Development

Website Github Doc Discord

Availability & Pricing

Code Access Open Source
Pricing Model
Free Custom Pricing
Details
Usage-based

AI Curation

Curator Agent updated description, category, subcategory, and 3 more

14 days ago