Using SQL in Your Data Lake with DuckDB, Iceberg, dbt, and MotherDuck

Video Description

In this talk at our MotherDuck Seattle meetup, Jacob, developer advocate at MotherDuck, talked about why we should use more SQL and go hands-on with some practical examples of how to read from Iceberg using dbt, DuckDB, and MotherDuck. ☁️🦆 Start using DuckDB in the Cloud for FREE with MotherDuck : https://hubs.la/Q02QnFR40 Check out our next events at https://motherduck.com/events/ ➡️ Follow Us LinkedIn: https://linkedin.com/company/motherduck X/Twitter : https://twitter.com/motherduck Blog: https://motherduck.com/blog/ #dataengineering #duckdb -------------------------------------- Discover how to leverage SQL across your entire data stack with this practical MotherDuck demo. We'll explore why DuckDB, a powerful in-process OLAP query engine, is becoming the SQL dialect that software developers love. Learn the fundamentals of DuckDB—a lightweight, parallel, Postgres-compatible engine—and see how MotherDuck extends it into a serverless cloud data warehouse, simplifying data workflows for data engineers, analysts, and developers. Get hands-on with the DuckDB CLI to query large datasets stored as Apache Iceberg tables. We'll walk through installing the necessary extensions to read Parquet files directly from an S3 data lake using a simple `iceberg_scan` SQL function. See how to `ATTACH` your local DuckDB session to MotherDuck, moving computation to the cloud for lower latency and faster queries on your S3 data. This tutorial is perfect for anyone asking "how to query Iceberg with DuckDB." Streamline your data transformation pipeline by integrating dbt with MotherDuck. Learn how to define your Iceberg tables as sources and materialize them as views, creating a schema-on-read layer. We'll then build a gold layer by materializing queries as tables in MotherDuck's native format, which can offer up to 5x faster performance. This demonstrates a powerful pattern for building efficient ELT pipelines using just SQL. We also showcase interactive analysis in the browser using DuckDB Wasm and temporary tables for zero-latency follow-up queries. Explore the incredible extensibility of DuckDB's ecosystem. See how to instantly create a JSON API for your data by loading the HTTP server extension and running a SQL query. We'll also demonstrate the G Sheets extension, showing you how to authenticate and write data directly from an Iceberg table or a MotherDuck query into a Google Sheet—a perfect solution for automating reports for business stakeholders. This highlights how SQL can be a powerful tool for both data processing and integration.