Zip vs Tar.gz Files Explained and Compared (Archiving and the DEFLATE algorithm)

Video Description

In this video, I explain the similarities and differences between two popular compressed archive formats, zip on Windows, and tar.gz in the *nix world. Both formats typically use the same compression, and also serve as a way to collect files together in an archive, however, there are a some fundamental differences between how they work and why they're used for different purposes. I discuss the two main steps of compression and linearizing the files into an archive, and how each stage differs between the two formats, as well as the advantages and disadvantages of each implementation. I hope you enjoy the video and learn something new! This is yet another video that is a bit rambly at times (having cut it down from over 30 minutes). I have added timestamps so you can easily skip between the sections you're interested in. I apologize in advance for the clipped (and heavily compressed) audio, this was recorded accidentally with a 10 dB boost on the mic, on a 128K MP3 recorder making for a questionable combination. ## Timestamps ### Introduction 00:00 - Introduction 00:19 - What is a zip and tar file? 01:45 - Gzip (and other compressors) 02:45 - Why use tar instead of zip on Linux? ### What's the difference? 03:48 - How does tar work? 04:39 - How does zip work? ### Comparison 05:09 - The advantages of zip 06:46 - The advantage of tar.gz (and streaming compression) 08:30 - The disadvantage of tar ### Further learning 09:30 - Notes on 7zip and .gz.tar 10:40 - Indexed tar files with pixz (and comparison to 7z) 13:30 - What should you use? 15:10 - Conclusion ## Links (Get Smarter Section) DEFLATE algorithm: Wiki: https://en.wikipedia.org/wiki/Deflate How DEFLATE works (good summary): https://zlib.net/feldspar.html Full specification: https://datatracker.ietf.org/doc/html/rfc1951 TAR format: Wiki: https://en.wikipedia.org/wiki/Tar_(computing) Man page: https://linux.die.net/man/1/tar Tar format specs: https://www.gnu.org/software/tar/manual/html_node/Standard.html Gzip: (based on DEFLATE) Wiki: https://en.wikipedia.org/wiki/Gzip Homepage: https://www.gnu.org/software/gzip/ Bzip2: Wiki: https://en.wikipedia.org/wiki/Bzip2 Homepage: https://www.sourceware.org/bzip2/ XZ utils: (LZMA2 compression) Wiki: https://en.wikipedia.org/wiki/XZ_Utils Homepage: https://tukaani.org/xz/ pixz: (parallel indexed xz) https://github.com/vasi/pixz pigz: (parallel implementation of gz) https://github.com/madler/pigz Lzip: (also based on LZMA2) Homepage: https://www.nongnu.org/lzip/ LZMA2 Compression: Wiki: https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Markov_chain_algorithm Z-Standard Compression: (aka zstd) Wiki: https://en.wikipedia.org/wiki/Zstd Homepage: https://facebook.github.io/zstd/ Source: https://github.com/facebook/zstd 7zip: (also generally LZMA) Wiki: https://en.wikipedia.org/wiki/7-Zip Homepage: https://www.7-zip.org/ Source code: https://sourceforge.net/projects/sevenzip/ p7zip (POSIX port): http://p7zip.sourceforge.net/ Zip: (generally DEFLATE) Wiki: https://en.wikipedia.org/wiki/ZIP_(file_format) Specs: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT Dar: (competing new format for tar) http://dar.linux.free.fr/ ## Content used: Zip and Tar icons in thumbnail from FlatIcon. Ending music is We'll Meet Again by TheFatRat ## Clarifications and Corrections Just to clarify a few things before I get some comments: The 'only decompress the file' benefit I mentioned in zip is because zip (and 7z) keep an index at the front. If you did .gz.tar, you wouldn't get that benefit, as tar isn't indexed. Next, when I say 'the index of the tar file is at the end', what I mean is that if you want the file list (like an index would produce), you need to read the archive to the end as though there is an index there (tar files don't have an index, just a few bytes at the front of each file). So, to get a file list, you need to read those bytes at the start of each file, meaning you have to read the full archive. I hope this clarifies it. Clarification: pigz is not indexed, pixz is. Pixz is backwards compatible with xz, although both support multithreading these days. (more to be added)