Tìm thấy 653 ứng dụng & công cụ phù hợp
Built a scalable cloud ETL pipeline using Python, PostgreSQL, Pandas, and Docker to efficiently process, clean, and load over 1.6 million records with optimized batch processing, fault tolerance, and
Spark-free Python utilities for Microsoft Fabric focused on Data Engineering using Polars and delta-rs
Example project demonstrating deployment patterns for real-time streaming workflows with Prefect 2.0
PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows
Companion repository to the ETL & ELT Pipelines with Apache Airflow® eBook
本项目旨在结合以往研究人员的代表性工作，从多个维度评估sft数据，并自动化过滤sft数据。
Python package for Customizable Data Preprocessing Pipelines
A Kubernetes Operator to orchestrate Benthos pipelines
Framework which helps you to make parallel/distributed calculations using data pipelines
SQL practice repository covering data analysis, joins, aggregations, window functions, and real-world query solving for data engineering and analytics.
rtdl makes it easy to build and maintain a real-time data lake
A DuckDB-powered command line interface for Snowflake security, governance, operations, and cost optimization.
Sonnet Scripts are a collection of pre-built data architecture patterns that you can quickly spin up on a local machine, along with examples of real-world data that you can use with it.
funsies is a lightweight workflow engine 🔧
A Pub/Sub for Tables based data integration platform, to discover, publish, modify and consume data effortlessly.
End-to-end ETL pipeline in the Microsoft Azure cloud - (Jun '24 - Jul '24)
data engineering 100 days 🤖 🧲 🦾 | #DE
AI lifecycle platform for classical ML and agentic systems. Versioned, encrypted registries for data, models, experiments, prompts, agents, MCPs, and skills. Pre/deploy/post-deployment patterns. Rea
Fast data transport : Apache Arrow, Protobuf and Message Pack data on the one stream. Measured faster than Arrow Flight on all open benchmarks. HTTP, TCP, QUIC, Websocket, Webtransport, Stdio, UDS, al
This project leverages GCS, Composer, Dataflow, BigQuery, and Looker on Google Cloud Platform (GCP) to build a robust data engineering solution for processing, storing, and reporting daily transaction
Four challenges involving data science knowledge and process automation using Python.
Next generation compute platform for the post-modern data stack
A complete LangGraph multi-agent system demo using SQL tools, Tavily search, MCP Toolbox, and OpenRouter models — with reproducible notebooks and a full supervisor-led agent workflow.
Pentaho plugin for Apache Airflow - Orquestate pentaho transformations and jobs from Airflow