Tableflow: Materialize Apache Kafka® Topics as Apache Iceberg™ and Delta Lake Tables With Zero ETL

Video Description

Blog post: https://cnfl.io/4hNOPFn | Tim Berglund is back at the lightboard to show you the most exciting thing he’s seen in analytics in the last 40 years (nope, that’s not an exaggeration): Tableflow, a new feature in Confluent Cloud that lets you completely skip the ETL and make your Apache Kafka® topic data available as tables in your data lake. In this video, Tim walks through how Tableflow works and why it’s a big deal—no data copying, no transformation steps, just native open table formats like Apache Iceberg™ and Delta Lake right on top of your Kafka data. Along the way, he explains how we got here (Hint: It involves the evolution of the data warehouse and a copious amount of wrestling with schema and pipelines over the years) and how Tableflow bridges the gap between operational and analytical systems, eliminating unnecessary data movement. Tim also explains how Apache Flink® fits into the picture to help you clean and prep your data upstream. If you’re working with Kafka and modern data lakes, this is one of those tools you’ll probably want to have in your back pocket. RELATED RESOURCES ► Tableflow Announcement: https://cnfl.io/4hNOPFn ► Tableflow Product Page: https://cnfl.io/4iXBGe2 ► Tableflow Feature Walkthrough: https://cnfl.io/423dEaR ► Try Confluent Cloud for Free: https://cnfl.io/440ZU2C CHAPTERS 00:00 – Intro to Tableflow 00:35 – Data warehousing: The classical era 02:50 – The rise of distributed data lakes 05:00 – ELT vs. ETL and the return of schema 06:40 – Emergence of open table formats 08:00 – Decoupling storage and compute in the cloud 08:45 – How data enters the lake today (Kafka) 10:00 – The cost of data copying and context loss 11:20 – How Tableflowworks: Kafka topics as tables 13:05 – How Tableflow uses Schema Registry 14:00 – Shift data transformation and cleaning left with Flink 15:45 – Try Tableflow in Confluent Cloud for free CONNECT Subscribe, if you dare: https://www.youtube.com/@ConfluentDeveloper?sub_confirmation=1 Community Slack: https://confluentcommunity.slack.com X: https://x.com/confluentinc Linkedin: https://www.linkedin.com/company/confluent GitHub: https://github.com/confluentinc Site: https://developer.confluent.io ABOUT CONFLUENT DEVELOPER Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io. #tableflow #apachekafka #kafka #confluent