A Metadata Service for Data Abstraction, Data Lineage & Event-based Triggers | WeWork

Video Description

Get the slides: https://www.datacouncil.ai/talks/marquez-a-metadata-service-for-data-abstraction-data-lineage-and-event-based-triggers ABOUT THE TALK: At WeWork, it's critical that we understand the complete context for all datasets. We also want to be able to explore dependencies between jobs and the datasets they produce and consume. To do this, WeWork needs metadata. In this talk I will focus on Marquez, a core service for the collection, aggregation and visualization of a data ecosystems metadata. Marquez maintains the provenance of how datasets are consumed and produced while providing global visibility into job runtime. ABOUT THE SPEAKER: Willy Lulciuc is a Data Engineer at WeWork and works on the Project Marquez team in San Francisco, making datasets discoverable and meaningful. Previously, he worked on the real-time streaming data platform powering BounceX, and before that, designed and scaled sensor data streams at Canary. When he's not reviewing code and creating indirections, he can be found experimenting with analog synthesizers. ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups. FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil-ai Facebook: https://www.facebook.com/datacouncilai Eventbrite: https://www.eventbrite.com/o/data-council-30357384520