필사 모드: Open Source Worth Watching Right Now (5) Data and ML Pipelines

• Introduction — the days when a single scheduler was enough are over
• Snapshot
• Ingestion and transformation
• Orchestration — what do you treat as the unit of management
• Execution engines
• Model lifecycle
• Search stores
• Before you adopt
• Links

Introduction — the days when a single scheduler was enough are over

For a long time, saying you were building a data pipeline meant registering jobs with a scheduler. Today that one sentence splits into six decisions. How to pull the source data in, where to transform it, what to treat as the unit of management, which engine to run the computation on, how to track model versions, and where to keep the embeddings.

A different project has settled into each of those decisions, and in many cases they are not competing with one another. What follows is not a ranking but a map organised by the slot each project fills.

Snapshot

All figures as of 2026-08-12.

Ingestion and transformation

dlt-hub/dlt handles the part where you pull data out of a source and load it into a destination, and it does so as a Python library. It takes on schema inference, coping with schema changes, and incremental loading, which cuts down on the near-identical code you used to write for every source. Being able to drop it into an existing codebase without standing up a separate service makes a real difference in practice.

dbt-labs/dbt-core pushed the practice of managing in-warehouse transformations with SQL and a dependency graph to something close to a standard. The essential part is that tests and documentation come attached to it. That said, computation outside the warehouse and real-time processing are not this tool's territory, and it is worth confirming where the line falls between the open source core and the commercial managed product.

Orchestration — what do you treat as the unit of management

dagster-io/dagster and PrefectHQ/prefect fill the same slot from different angles.

Dagster treats the output as the unit of management. Because the structure itself shows which table came out of which inputs, lineage tracking and partial re-runs come naturally. In exchange, you have to learn the concepts first.

Prefect is closer to attaching markers to existing Python functions and turning them into flows. The barrier to entry is low, so you can move scripts you already have across quickly.

Neither one should become an end in itself. If you have only a handful of jobs and the dependencies are simple, cron plus a well-written script may still be the right answer.

Execution engines

pola-rs/polars is a library that rewrote dataframe processing from scratch. It builds a query plan through lazy execution and then optimises it before running, so the ceiling on the data volume you can handle on a single machine rises noticeably.

# Example: filter and aggregate a large CSV with lazy execution
import polars as pl

q = (pl.scan_csv("events.csv")
       .filter(pl.col("status") == "ok")
       .group_by("country")
       .agg(pl.len().alias("n")))
print(q.collect().head())

apache/datafusion is not an end-user tool but a component for building query engines. Unless you are a team building a data product yourselves, you will rarely have occasion to pick this project; instead, there is a good chance it is already inside a tool you use.

ray-project/ray distributes Python work across multiple machines. It is used for workloads that go beyond one machine, such as training, tuning, and batch inference. Conversely, putting a cluster under a job that a single machine handles fine only raises the difficulty of debugging.

Model lifecycle

mlflow/mlflow is effectively close to the default for experiment tracking and the model registry. For a team that cannot answer which model, built from which data and which parameters, is currently in production, this is the first tool they need.

bentoml/BentoML takes on the side of packaging that model as a service. Its last push at the time of checking was 2026-08-03, a little less frequent than the other projects on the list. That does not mean anything is wrong, but I recommend looking at the release cadence and issue responses yourself and judging from there.

Search stores

qdrant/qdrant is a vector database that handles filters and vector search together. Its strengths show when you have a lot of searches that combine metadata conditions.

lancedb/lancedb chose the path of running on top of a file format without a server. You can start with an embedded setup, so the initial cost is low.

That said, if your data already lives in a relational database and the scale is not large, it is more sensible to try the vector extension of your existing database before bringing in a dedicated store. The decision to add one more system is hard to reverse.

Before you adopt

Check the full licence text yourself, and route commercial adoption through legal review. This post is not legal advice.

Everything on this list appears to be permissively licensed, but with data tools, many companies keep an open source core alongside a commercial managed product. You need the habit of checking whether a feature you saw in the documentation is a feature that lives inside the repository. And pipeline tools, once they are in, tend to stay for a long time. Start from the simplest setup that fits the size of your team today.

Repository details (stars, licence, recent activity) were checked directly on GitHub on 2026-08-12 and are point-in-time values. The numbers and the status change.

Links

Series: Previous post — Observability and security · Next post — How to evaluate an open source project

Related posts on this blog:

• Data platforms and the lakehouse, end to end
• MLOps feature stores and Feast
• Vector database comparison

Tools: CSV and JSON converter · SQL playground