cloud-etl-pipeline

Data-Engineering Open Source

Built a scalable cloud ETL pipeline using Python, PostgreSQL, Pandas, and Docker to efficiently process, clean, and load over 1.6 million records with optimized batch processing, fault tolerance, and

★ 45 Stars ⑂ 1 Forks License: MIT License

Download Latest Release Official Website ↗ GitHub Repository ↗

About cloud-etl-pipeline

Built a scalable cloud ETL pipeline using Python, PostgreSQL, Pandas, and Docker to efficiently process, clean, and load over 1.6 million records with optimized batch processing, fault tolerance, and automated data validation.

Supported Platforms

Open Source Web Cross-Platform

Related in Data-Engineering

data-engineering-zoomcamp

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼

prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

airbyte

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

taipy

Turns Data and AI algorithms into production-ready web applications in no time.

dagster

An orchestration platform for the development, production, and observation of data assets.

Cookbook

The Data Engineering Cookbook