Skip to content

Aqueducts

Aqueducts is a framework to write and execute ETL data pipelines declaratively. Define your data processing workflows in YAML and execute them locally or remotely with built-in cloud storage support.

Key Features

  • Declarative Pipeline Configuration: Define ETL pipelines in YAML, JSON, or TOML
  • Pipeline Execution: Run pipelines locally, remotely, or embedded in your applications
  • Data Source Support: Extract from CSV, JSONL, Parquet files, Delta tables, and ODBC databases
  • Data Destination Support: Load data into local files, object stores, or Delta tables
  • Cloud Storage Support: Built-in support for Local, S3, GCS, and Azure Blob storage
  • Advanced Operations: Upsert/Replace/Append operations with partition support
  • Template System: Parameter substitution with ${variable} syntax

Getting Started

Ready to build your first pipeline? Head to Getting Started for installation and a quick tutorial.

Examples

Browse real-world pipeline examples in the examples folder on GitHub.

Community

Join our Discord community to get help, share your work, and connect with other Aqueducts users:

Discord