How to Deploy Dask | Deployment Workshop | Dask Summit 2021

Video Description

This workshop will cover the most common methods for how to deploy Dask today. The speakers are Jacob Tomlinson, Adam Lewis, Amit Kumar, Anderson Banihirwe, Brad Miro, and Nanthini Balasubramanian. Starting with an overview of all the moving pieces within a Dask cluster (client, cluster, scheduler, workers), we will then talk through various platforms and the tools used to deploy onto them along with benefits, common challenges, and pitfalls. This workshop will start out by covering how a Dask cluster is deployed at a low level with manually running a scheduler and some workers and then connecting a client. We will walk through how these pieces find each other and communicate, and how we can find out more information ourselves using the dashboard and utilities in Python. We will also cover at a high level the three types of deployment that folks tend to do; fixed where a single Dask cluster is installed and run indefinitely, ephemeral where clusters are created and destroyed as part of a single workflow and multi-tenant where clusters are created and managed centrally by Dask Gateway. We will also have a series of talks to give an overview of platform specific tooling. These talks will outline how to deploy Dask clusters on Kubernetes, HPC, Cloud, and with Dask Gateway as well as dive into the strengths, challenges and limitations of each platform. By the end of the workshop, attendees will have an understanding of how Dask clusters are deployed and communicate in general along with some relevant examples for how they can deploy Dask and scale it on their infrastructure. KEY MOMENTS 00:00:00 - Intro 00:03:00 - Jacob Tomlinson 00:33:00 - Anderson Banihirwe 00:39:00 - Brad Miro 01:04:00 - Nanthini Balasubramanian 01:16:50 - Amit Kumar, Adam Lewis 01:45:00 - Closing What is the Dask Summit? The Dask Distributed Summit is where users, contributors, and newcomers can share experiences to learn from one another and grow together. The Dask Distributed Summit provides content, information, and learning opportunities for attendees of all levels of Dask familiarity and expertise. Share your feedback with us in the comments and let us know: - Did you find this workshop on how to deploy Dask helpful? - Have you used Dask before? Learn more at summit.dask.org and dask.org