The Open Software & Tools Directory

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

Turns Data and AI algorithms into production-ready web applications in no time.

An orchestration platform for the development, production, and observation of data assets.

The Data Engineering Cookbook

Roadmap to becoming a data engineer in 2021

Always know what to expect from your data.

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

Open Source Feature Flags, Experimentation, and Product Analytics

Business intelligence as code: build fast, interactive data visualizations in SQL and markdown

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

Privacy and Security focused Segment-alternative, in Golang and React

SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence. This project is 100% free and open source.

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Par

A list of useful resources to learn Data Engineering from scratch

Blazing-fast Data-Wrangling toolkit

An Awesome List of Open-Source Data Engineering Projects

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

Data Contracts engine for the modern data stack. https://www.soda.io

Implementing best practices for PySpark ETL jobs and applications.

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Feathr – A scalable, unified data and AI engineering platform for enterprise