lightstream

Data-Engineering Open Source

★ 41 Stars ⑂ 1 Forks License: Other

Download Latest Release Official Website ↗ GitHub Repository ↗

About lightstream

Fast data transport : Apache Arrow, Protobuf and Message Pack data on the one stream. Measured faster than Arrow Flight on all open benchmarks. HTTP, TCP, QUIC, Websocket, Webtransport, Stdio, UDS, all interchangeable, with 64-byte SIMD compatibilty maintained over the network and to/from disk via custom Arrow Mmap and Parquet Readers and Writers

Supported Platforms

Open Source Web Cross-Platform

Related in Data-Engineering

data-engineering-zoomcamp

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼

prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

airbyte

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

taipy

Turns Data and AI algorithms into production-ready web applications in no time.

dagster

An orchestration platform for the development, production, and observation of data assets.

Cookbook

The Data Engineering Cookbook