BU
Back to BlogAll Articles
PipelinesPostgreSQLDelta Lake
Building an Automated Ingestion Pipeline from PostgreSQL to Delta Lake
LS
Lakehouse Studio Team
May 20, 20262 min read
Building an Automated Ingestion Pipeline from PostgreSQL to Delta Lake
One of the most common data engineering tasks is keeping a Delta table in sync with a transactional PostgreSQL database. This post walks through doing exactly that using the Lakehouse Studio pipeline wizard.
The Setup
We have a public.orders table in PostgreSQL that receives new rows throughout the day. We want to merge these into a Delta table main.sales.orders in our lakehouse, deduplicated on order_id.
Step 1 — Register the Connection
- Go to Connections → New Connection
- Select PostgreSQL
- Enter host, port, database, username, and password
- Click Test Connection — you should see a green check
- Click Save
Step 2 — Create the Pipeline
- Go to Pipelines → New Pipeline
- Source: select your PostgreSQL connection, enter
public.ordersas the source table - Destination: choose catalog
main, schemasales, table nameorders - Write Mode: select
merge, set merge key toorder_id - Schedule: select
Hourly - Notifications: enter your email to get alerted on failure
Step 3 — Run It
Click Run Now to trigger the first load. Switch to the Runs tab to watch the status. You will see rows ingested, files processed, and duration.
Checking the Results
Open the SQL Workspace and run:
SELECT COUNT(*) FROM main.sales.orders;
Your data is now in a Delta table, queryable via Spark and available to any connected BI tool.
Published by
Datrixen Analytica