Jim Crist - Make it Work, Make it Right, Make it Fast Debugging and Profiling in Dask

Video Description

Description Dask is a pure python library for parallel and distributed computing. It's designed with flexibility in mind, making it easy to parallelize the complicated workflows often found in science. However, once you get something working, how do you debug or profile it? In this talk we'll cover the various tools Dask provides for diagnosing bugs and bottlenecks, as well as tips for resolving these issues. Abstract Dask is a pure python library for parallel and distributed computing. It's designed with simplicity and flexibility in mind, making it easy to parallelize the complicated workflows often found in science. However, once you get something working, how do you debug or profile it? Debugging and profiling parallel code is notoriously hard! In this talk we'll cover the various tools Dask provides for diagnosing bugs and performance bottlenecks, as well as tips and techniques for resolving these issues. Starting with an example single-threaded probram, we'll walk through adding Dask to parallelize it, and then iterate on this example to gradually improve performance throughout the talk. Attendees should leave having a better understanding of: -What tools Dask provides for debugging on a single machine -How Dask uses IPython to make debugging distributed computations easy -How to profile Dask code both on a single machine and on a cluster -How to interpret the graphs presented in the Dask Dashboard -Performance techniques for attacking bottlenecks once they've been identified. www.pydata.org PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R. PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases. 00:00 Welcome! 00:10 Help us add time stamps or captions to this video! See the description for details. Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps