So friends, to kick us off, the very first thing that I want to talk to you
about is the concept of compression as well as archiving files on Linux.
Now quite honestly, very often people do confuse these different concepts, so
it is important that we have some clarity before we begin automating these
particular tasks.
Now ultimately, when we happen to invoke compression and archiving, we can
basically use them independently or we can combine them together,
but really at its core when we're talking about compression, we are talking
about making files smaller. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
We are, after all, squishing them and compressing them into a smaller footprint
.
This is really what we're talking about.
Whereas with respect to our archiving of files, what we're doing is essentially
taking multiple files, or perhaps maybe many directories and subdirectories,
and essentially grouping them together to be one particular file.
So effectively, when we happen to combine the archiving of files with
compression,
we can take many different files and many different directories and combine
them into one particular file,
and then we can further compress that individual file as we see fit and then WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
and then we can further compress that individual file as we see fit and then
move it around as we want.
Moving around, simply meaning that the file can be more portable as opposed to
having many different large directories and files,
you could have one small compressed file that you could perhaps, for example,
just email over to someone or upload to a particular file sharing site, that
type of thing.
So with respect to compression, there are three main utilities that we have
available to us on Linux systems.
We have a compression utility known as Gzip, another known as Bzip2, and WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
We have a compression utility known as Gzip, another known as Bzip2, and
another known as XZ.
Now, like I said, each of these tools allow us to compress files, and they also
have corresponding tools to decompress or uncompress the files back to the
normal state,
so Gzip can be uncompressed with something known as Gunzip, and Bzip2 can be
uncompressed with Bonsip2, and XZ can be uncompressed with OnXZ.
Now, you might be thinking, why on earth do we have all these different
utilities to effectively do the same thing, all of these tools on the left here WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
utilities to effectively do the same thing, all of these tools on the left here
,
well, they all effectively make files smaller after all, and the ones on the
right ultimately return the files back to their original state.
Now, quite honestly, we don't have to get two into the weeds about the
different types of compression and the encoding used, that type of thing,
but ultimately at its core, different compression ratios can produce smaller
files, for example, have better compression ratios,
and others, whilst not maybe compressing quite as much, can do the job faster,
whereas others yet again, maybe just provide a happy medium between WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
whereas others yet again, maybe just provide a happy medium between
aggressively compressing the file and performing the task very, very quickly.
So Gzip is a very, very common implementation, what it tends to do is it
provides fairly good compression, but it does so very, very quickly.
So if you just want to maybe compress files and you're not too concerned about
the actual size or the resultant output,
you just want to get the job done quick and easy, Gzip2 can be a good solution
for you, whereas Bzip2 happens to produce smaller files, particularly when you
're dealing with text files.
Ultimately it requires more CPU than Gzip and it's going to take more time to WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
Ultimately it requires more CPU than Gzip and it's going to take more time to
perform a particular task.
Now, if you just happen to be compressing just a few files that are not so
large, the difference in speed is going to be absolutely negligible,
we're talking about compressing at scale here, and then the latest one, XZ
compression.
This typically produces the best compression ratio of the three, but it does
seem to depend on the actual data that you're trying to compress.
If for example you're just trying to compress a small text file, then you may
notice that Bzip2 actually produces a smaller footprint, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
notice that Bzip2 actually produces a smaller footprint,
whereas dealing with very, very large datasets, you may notice that XZ is in
fact the best option for you.
Now like I say, we don't want to go into too much detail about these different
compression ratios, we just want to have a general understanding of the
different options that we have available to us before we begin to automate
things.
With that said, let's dive in and use these tools to get a simple example of
what we're trying to do here.
So what I will do is I will create a little text file here, and I'll just call
this "File1.txt" because as you know, I am super duper original. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
this "File1.txt" because as you know, I am super duper original.
Now just to get some text, I've just googled for Ansible on Wikipedia and I'm
just going to grab all of this text here within this page.
It's not a huge amount of information, but it will do for the demonstration.
So I'll just paste all this in and I'll just press WQ to write and quit.
Now if I do an ls-l on the file1.txt, we can actually see the size of the file,
in fact, if I just happen to do a -h, we can see it as human readable, we can
see this as 13 kilobytes.
So let's try to compress this particular file. If I happen to say gzip and I WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So let's try to compress this particular file. If I happen to say gzip and I
target file1 and just hit enter, notice the file has actually been converted to
have a .gz extension.
This tells us that it has been compressed using gzip. Now crucially, if I do an
ls-l on the file.txt.gz now and hit enter, notice the file is only 5.2k in size
as compared to before when it was 13.
So clearly we can see here this particular file is less than half the original
size.
Now in order to return this file back to its original state, we can say gunsip WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
Now in order to return this file back to its original state, we can say gunsip
and then target the actual file right here.
If I now do an ls, the file is back to its original state. And as we can see
here, if I just do an ls-l on the file, it goes back to its original size of 13
.K.
Now if I happen to say bzip2 on the file and to enter and I now do an ls,
notice the file now has a .bz extension. This tells us we're using bzip2
compression and if we want to uncompress this file, we would use gunsip2.
But before we actually do this, let's assess the new size of this particular WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
But before we actually do this, let's assess the new size of this particular
file if I hit enter. Notice the file is actually a little bit smaller than we
got with gzip compression.
So like I say bzip2 gives us a smaller footprint but generally speaking it's
going to take more CPU and a little bit longer to perform this compression.
In the case here where we have one small file, the difference is absolutely
negligible in terms of performance.
So if I want to uncompress this, I just say bzip2 and then go into the file and
to enter, if I do an ls, the file has returned and I get an ls-lh. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
to enter, if I do an ls, the file has returned and I get an ls-lh.
On the file we can see it goes back to its original state and then at the very
end if I just say xz on file1.txt, do an ls, we get the file once again and it
has this time the .xz compression.
Now like I say with respect to xz on small text files, you're not going to
notice any better performance versus bzip2 or rather bzip2 should I say.
Notice it's actually a little bit bigger but like I say with very very large
datasets of different data types, xz tends to actually outperform bzip2 in
these scenarios. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So again how do we uncompress in this case here, we just say on xz on file1.txt
, do an ls on the file and the file has returned back to its original state.
Now check this out. Let's go and get some more text here.
Okay so I've just googled for red hat on Wikipedia. Let me just grab all of
this stuff here. Let's create a new file called file2.txt.
So let's paste this in. There we go. Let's wq to write and quits.
So now I have file1 and file2.txt. Now if I happen to say gzip on file1 as well WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So now I have file1 and file2.txt. Now if I happen to say gzip on file1 as well
as file2.txt, watch what happens and if I do an ls-lh, the files have actually
been compressed but notice they happen to be compressed separately than not one
file.
So if I wanted to compress a whole directory of files and perhaps maybe share
them, now we'd have to effectively upload multiple different compressed files
as opposed to a one singular file.
Similarly, let me just show you an example here. Let's create a directory
called example. And I'll just guns up these two files.
So now they're back to the original state. If I move these files into the WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So now they're back to the original state. If I move these files into the
example there and I now try to say gzip on the example directory and to enter,
notice here that says we can't actually compress a directory itself.
So if we go inside the directory and do an ls, the files are still in their
original uncompressed format.
So this is now when we begin to talk about archiving particular files. Like I
said earlier, this is going to allow us to group the files together as a single
file.
Now in this case here, the utility that we want to use is one called tar. This
is known as a tape archive.
Now like I say, this particular utility by default is not going to actually WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
Now like I say, this particular utility by default is not going to actually
compress the files but it can actually do so with particular options.
So let me go and show you how this operates now. So let me just move these
particular files back in directory.
So if I do an ls, I'm no longer here. Go back in directory and try that again.
Go back in directory. There we go.
Now we can see they're back in the main home door. Check this out now. Here's
the syntax.
If you happen to look at the man page for the tar commands, if you press space
bar, you're going to see all these different switches such as -c to create a new WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
bar, you're going to see all these different switches such as -c to create a new
archive, -x to extract the archive back to its original form,
and -f allows us to specify the file that we want to archive or unarchive.
Now just one point, when we happen to create an archive, this is known as a tar
ball, which is the new singular file that we have created, which is comprising
of the multiple files within.
And like I say, this is not going to be compressed, but if we want to add the -
j flag to our tarball, then we will effectively filter the archive through the b WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
j flag to our tarball, then we will effectively filter the archive through the b
zip to compression.
Whereas if we use -g we will filter the archive through xz compression, and if
we use lowercase z, we will filter the archive through the gzip compression.
You see that?
So let's actually do this then. So what I will do here is I'll say tar, -cf to
create a file, and let's just call this my archive.tar, okay?
And now we'll specify all the files that I want to belong within this archive,
so I'll say file 1 should be part of it, as well as file number 2. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
so I'll say file 1 should be part of it, as well as file number 2.
So if I enter to an LS, notice the original files actually stay here, they've
not been converted directly, but we also have this copy known as myarchive.tar.
Now check this out, if I happen to remove these individual files that we use to
create the archive, so all that we have left is the myarchive.tar, and if I do
an LS-LH on the actual archive itself,
we can see the size is 60k. Now if I happen to uncompress this, I can say tar -
cf to extract the file on my archive.tar, notice the files have now been WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
cf to extract the file on my archive.tar, notice the files have now been
returned to the directory, and the archive still actually remains, you see that
?
Now if I happen to remove my archive.tar, and I once again say tar, and I'll
say -c to create -lhkj to run through bzip2 compression, and then if, and I
will create an archive called my compress archive, okay?
And I'll just say -tar, and I'll also add on the bz to extension, just to let
people know that the file has been put through the bzip2 compression, and now I WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
people know that the file has been put through the bzip2 compression, and now I
specify the files that I want to belong to this archive, so again, file 1 and
file 2.
If I hit enter, do an LS, we've now created this new archive right here, if I
do an LS-LH on my compressed archive, notice before it was 60k in size, whereas
the compress version is only 18k in size, much much smaller, and again, if we
want to uncompress, we can just say tar -x, j, f, so extract the file with bzip
2 compression, and then we specify the name of the file again. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
In fact, before I do that, let me just remove the individual files that we use
to make it, so that all we have left is the actual archive.
So then I say tar, x, j, f, on my compressed archive, and hit enter, do an LS,
now the files are back again, you see that?
Cat file 1.txt.
So that is the main operations behind our archiving and compression on Linux,
we have different options that we can use to compress the files with different
compression ratios, with different rates of performance, and when we want to
group together multiple files, as one which can either be compressed or uncomp WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
group together multiple files, as one which can either be compressed or uncomp
ressed, we want to use the tar utility.
So to understand the differences between these concepts, let's look at how we
can use Ansible to automate these particular tasks.