HPC kitchen [data-management]: Data management is as important as data storage

Video Description

People may think that getting a big enough storage space for your data is all you need. There's far more than that. It's easy for data (and kitchens) to become a huge mess without some care. It's even worse when you work together. Unfortunately, it's far too common for people to be too rushed to organize their stuff well, leading to worse problems down the line. Take care of your data and you data will t̶a̶k̶e̶ ̶c̶a̶r̶e̶ ̶o̶f̶ be valuable to you. Extra comments: * This is only a small view into all the ways that data management can go wrong. I can't tell you the solution to your problem, but you are smart and once you realize it, you can probably find the solution for you. * Don't forget about the safety of storage. If you have your data somewhere that isn't backed up, there's a high chance of losing it someday. It's best to use professionally managed systems when you can. 00:04 Data management isn't just data storage 00:40 Reminder our jargon in our HPC metaphor 00:01 Common problem: disorganization with multiple people 01:53 Common problem: can't share data when collaborating 03:00 Common problem: can't share because of confidential data mixed with shareable 03:37 Common problem: can't interpert the data 04:08 Common problem: non-standardization 04:56 Common problem: no long-term management 05:44 Common problem: Unclear permissions, not documented 06:16 Common problem: Code not reproducible 06:45 Common problem: not archiving things 07:23 Examining the patterns and common solutions (clear organization, clear usage rights, tell everyone the standards, ) 08:10 The biggest barrier is time and short-term rush. We need to work together on this 08:44 We must have time to do data management well, supervisors need to give us time to do this ----- Playlist: https://www.youtube.com/playlist?list=PLZLVmS9rf3nNDHRo1Baz_JVQWDI0mTYyB