Welcome back to vSphere 8 in the real world, and now we're going to discuss security logging.
So, logging is obviously a pretty important thing. Oftentimes, if you're having some kind
of issues, you might see something is failing, and messages that you get from the vSphere
graphical environment might not be very detailed. They might not really give you a good insight to
what's happening. You'll see something fail, but you don't really necessarily know why.
So, your log files are going to be a great resource. Sometimes, the logging will have to be
enhanced to be more robust than what you get by default. You don't want to always log in a very
verbose mode because that can start to create a lot of data, a lot of log files, a lot of space
gets used, and it can be a lot of noise. So, when you get too much logging information on a regular
basis, it may be hard to find the items that are actually relative to what you're trying to do.
And again, it's just a lot of space and things like that. Now, if you're trying to troubleshoot
a specific problem, then it might be very relevant to go in and increase the logging level to a more
robust level, maybe even like the most robust level, while you're trying to troubleshoot this
problem to get that information. And you also probably have an idea what you're searching for
to kind of make it not such a needle in a haystack, right? If you're just kind of doing
general observation of the logs, not really trying to troubleshoot any specific issue,
then a huge amount of information is just overwhelming. But if you know I'm looking
for information about a certain virtual machine or about a certain VM kernel port or something like
that, about the HBA not functioning properly or something, then you can start narrowing down your
search and start going through that huge amount of data. Once you've got your information,
you've resolved your issue, you're past the troubleshooting and implementing your solution,
and you're back to a status quo, then you can lower your logging level back to a more normal
type of logging level. So, logging is super important. If you want to troubleshoot things,
if you're looking for any kind of security warnings about unusual traffic behaviors,
things like that, and you might have even legal records maybe. I don't have a lot of experience
legal records unless it's like for the insurance company during a ransomware remediation or
something like that, that will come in for the insurance company where they want logs and things
like that from what's happening. If you have no logging enabled, you can turn logging off.
Obviously, not a great idea, but you can. Then all of a sudden your insurance company might
start to get suspicious and things like that if you ever need to go to them, and that is becoming
something more and more common. So, where your log files are stored,
if you're searching for log file information from vCenter, then it will be stored on that
vCenter virtual appliance. Log files for virtual machines are going to be stored on the data store
with the virtual machine, and your ESXi host will have its own log files. Not that long ago,
I had a client contact me, and their vCenter was having all kinds of problems and issues,
and the hard drive was full, and they couldn't figure out what was going on.
Come to find out their log files were filling up their vCenter server, and I said, well,
why don't we look at sending your log files to a syslog server? And they, oh, we have a syslog
server. Come to find out their syslog server had inadvertently been shut down. And so,
the vCenter server was just sitting on all these log files it was creating, and they had a pretty
robust logging level set. Not the most, but pretty robust. And once we powered on the syslog server,
and we got it, you know, to reestablish communication with it, and started sending
those log files to the syslog server, all of a sudden, all their instability issues with vCenter
went away. And they were logging, you know, fully again and things like that, and they didn't even
realize that their syslog server wasn't up and running. So, that's another conversation,
another kind of scary issue there. But syslog servers are a great resource. Not only will
allow you to send all your vCenter, you know, files there for your logging files there,
you can send your vCenter log files there, you can send your user client host log files there,
you can send all your, you know, systems log files to this one place. And if you're looking
for a problem, you can say, I want to find all the information, you know, what's happened from
vCenter and on these hosts during this time, during this 30-minute window. If you know,
you know, something happened that time, and you may not know exactly where the problem
happened at, but if you can say, okay, if I know what happened during this window here,
I'm going to search for 15 minutes before and 15 minutes after, and try and find all this,
maybe you want to do 20 minutes before, 20 minutes after, whatever, then you're getting,
oh, you know, vCenter did this, and the host did that, and this other host did this,
and vCenter did that. And so, you can kind of sift through all that. If you don't have a syslog
server kind of, you know, aggregating all this information together, then what ends up happening
is you have to go to vCenter and look at that timeframe, then go to the host and kind of look
at the timeframe, and go to another host and try and do that manually. And it can be very,
very difficult to do. It's great when you can kind of have all that information aggregated.
So, logging is super important. Definitely recommend you do, you know, you do enable logging.
Don't get too carried away with your logging. Don't, you know, enable the logging level to be
too high unless you're troubleshooting this specific problem that that logging level calls
for. And then when you're done troubleshooting that problem and you have your issue resolved,
put your logging level back to a more appropriate level for your environment. All right. So,
how can we enable logging? We can enable logging on a virtual machine level. We can also enable
logging for our ESXi host. We can also enable logging for our vCenter. The two places that
we're going to look at primarily will be our ESXi host, you know, checking our logging level for
that, and then saying, you know, how can we tune that logging a little bit for our VMs.
And then we can also enable logging or, you know, tune logging for vCenter. All right.
So, some logging options. Now, these levels here we're looking at, this is vCenter specific. So,
this is not the levels you'll have for a virtual machine. But when we're talking about logging for
vCenter, and vCenter is where a lot of information is gathered, right? Communication with the
different VMs. If you try to do a vMotion, the host will have information about that in their
logs, but vCenter will have information about, you know, initiated a vMotion from host one to host
two, and this is what happened, and this is what the success and failure was, and things like that,
right? So, the logging information from vCenter is going to be really, really important.
So, you can say you want no logging at all, or you can say only errors. So, very obvious, like,
this failed, this stopped type errors. Warnings are a little more detailed, right? And as you can
see, as they go down, it becomes more and more detailed, right? So, warnings are more detailed
rather than be at the hard critical, like, you know, a red stop, like more of a yellow level
type of situation where we're seeing, like, slow response times. CPU is not pegged at 100%,
but we're creeping up at 85%. You know, that would be a warning. For information, that would
be errors, okay? Again, a little more information. So, not necessarily, like, a hard failure, but,
like, something you might want to, like, watch out for, that kind of stuff. Verbose is going to be,
again, more detailed, and that's going to give you a lot of information. And trivia means, like,
even trivial things are being logged. So, trivia means every single thing. That's going to be some
very, very detailed logging. You might kind of consider it more like, or another term you'll
see in other vendors is a debug, right, where they just give you every single thing,
every little detail. And a lot of the information you see may not be an error. It may not be,
you know, something failed. It's something happened successfully, even, you know,
and here's all this information, every little thing that happens. And you might want to say,
well, this succeeded, and this succeeded. Not just, well, this failed and this failed,
but what happened between these two things failing? Well, this failed, these five things
happened successfully, then that failed. And that kind of, oh, I see why this failed,
because this and this. So, that trivia will give you a huge amount of information, which means
huge log files, huge information being sent to your syslog server. Some syslog solutions will
charge you based upon, like, the gigabyte, like how much is being, you know, ingested every day.
And so, if you ran this trivia every day, all day, then you'd be paying a ton to your syslog vendor.
So, you might want to just do errors or maybe warnings at the most, right? But when you're
troubleshooting, trying to, you know, dig into a problem that you just haven't been able to resolve
with a warning or information, you might go, you know, as far as verbose or trivia.
All right. So, this is, you know, the further down you go, the more details you get.
Most people are pretty comfortable for their status quo being errors and or warnings only.
All right. Those are some of your logging options there.
All right. So, we discussed how to enable logging and some logging options. Let's go ahead and go
into our demo environment. I'll show you where you can enable logging for a VM, where you can
enable logging for vCenter, and a couple of examples of what you might want to configure
for some advanced options for your VM logging. So, if you want to configure logging for your
vCenter, we are here in our vCenter server. I'm going to click on this vCenter,
and I will want to go to my configuration.
And right under general, you see I have my logging options here, right? So, I'm going to go
ahead and I see that my logging levels are set to info. I'm going to click on edit here
and go to logging settings. And right now, I'm set to info. I have, you know, none, error,
warning, info. I can go to verbose or to trivia. So, I can say, you know, which of these. Info is
probably fine. A lot of people are going to be okay with errors or warnings, right? Info me at
the most. You wouldn't want to go as far as verbose on an ongoing regular basis just because
that, again, can become an overwhelming amount of information. But, you know, one of these three
is usually where most people are going to land. Again, you're not going to do none. Error is the
least amount and still doing some logging, and that might not be enough. You might not want
only errors. You might want some warnings. So, before something becomes a problem,
when you see, like, utilization on something creeping up, you might want that warning,
right? You might not want necessarily just to get blindsided by a failure. You want a warning
something's coming, right? That kind of thing. So, you can go in and change your logging level
for your – this is for vCenter specifically, right? This is how we can enable and change our
vCenter logging is from here. We can also, for example, enable vCenter logging or, you know,
modify logging for a VM. I can go to this VM, and I can go to my edit settings. I'll go to VM
options, go to advanced here, right? And I can say enable logging, and I have a couple of different
options, right? For a VM, I have – I want, you know, run normally record debugging information
or record statistics. So, just some basic, you know, what I want to record here. I have some
options for that. Now, some things we can do is we can also go in and specify, like, how long do I
want to retain log files for? Basically, what I can say is how many log files do I want to retain,
how big can my log files get, that kind of thing. To configure that, I'm going to go to my advanced
parameters. And I could look to see do I have anything here. Now, one thing you'll notice real
quick, though, before I go and start looking at that, I can't go in there, and I can't change
this. The reason why I can't change anything here is this VM is running. So, if I hit cancel,
this VM is running. I'm just going to quickly just power this off. I probably should do a
shutdown, but this is a lab. I just want to be quick. I'm going to say power it off.
Now that it's off, if I go in and say edit settings and go to my advanced parameters,
now you see I can click in here. I can make changes. So, one, like, one thing you might
look for is do vmx.log and see if we have, do we have anything in here. So, nothing there.
So, I made some notes over here. Oops. Here. So, here are some options here for
this vmx.log.keepold. This is going to say how many log files do I want to keep. The default,
if there's no parameter set here, it's going to be 10. I can go in there and say make it 10. So,
now it's, like, you know, hard-coded. Or I can say, you know what, I have a lot of storage space,
and logs are really important to my organization. So, I want to keep 20, and I can add that,
right. So, now if I do a search for vmx, I have my vmx.keepold and a value of 20, right.
I can also, another option is rotate size. By default, if we don't configure this, like,
right now, that is not configured. What will happen is the system is going to create a new
log file every time a VM is powered on. It's removed from being suspended. It gets vMotion
to a new host. All these different things can cause a new log file to be created.
If I want to, I can say I want it to reach a certain size. Rotate logs when it reaches a
certain size. So, this is my rotate log file size, and I believe it's measured in bits. So,
the numbers are going to be some really huge number. I think I actually have a note of it
right here. Like, this right here would be about 2 megs, right. So, you can see it can get to be
a little bit cumbersome to try, because this is just in bits. So, if we wanted it to be,
you know, instead of 2 megs, we want it to be 20 megs. We can add a zero onto there,
and say add that, and then all of a sudden our log files can start to get bigger and bigger.
By default, it's going to just rotate, though, when we have a new event happen on the VM,
like a power on or a migration or being unsuspended from, you know, a suspended state.
So, under your advanced parameters, when the VM's off, we can go in and we can customize
these advanced options. For example, how many log files. The default being 10. We can make it
whatever we need to, as long as we don't get too carried away and start filling up our hard disk
space. We don't want to crash VMs because we filled up our SAM of log files.