Welcome back to vSphere 8 in the real world, and now we're going to discuss SR-IOV.
So SR-IOV is Single Root I-O Virtualization, and it's actually a pretty cool feature.
It's something that does require compatible hardware to do.
You need compatible network adapters specifically for this to work.
So the way virtual networking normally will work is you have your physical adapter
and a packet comes in to the adapter.
Now, the adapter might be able to read the MAC address, but in order to process that particular packet,
there's going to be an interrupt sent from the network card to a core on the CPU to get processing power
to then go through, process that packet, and then what's going to happen,
it's going to go through the entire VM kernel networking stack there.
It's going to go from the adapter, through the PCI bus, to the kernel, through the virtual switch,
through the port group, reprocess all the policies for that port group, and then to the VM.
And the VM is going to need a core, an interrupt from the core, to accept that packet,
and the network card needs a core or an interrupt to process that packet.
And so going through that system is going to actually use quite a bit of CPU time.
What the single-root IO virtualization does is it's a network card that has the ability to create these queues,
and the queues can be associated with certain virtual machines,
and then the virtual machines are able to get the packet directly from the adapter.
The adapter can write into the VM's RAM, and now the VM has that packet,
without having to do a bunch of additional processing by the physical processor,
by our Intel Xeon whatever processor.
We're not having to send as many interrupts through the system.
The actual network card handles receiving the packet, processing the packet,
and then putting it into RAM on that virtual machine.
So we need to have host to support it, meaning we need to have a version of ESXi that supports it.
We need to have physical NICs that support it.
Then we need to install a driver, a specific driver, an SR-IOV or single-root IO virtualization,
SR-IOV network adapter on the VM, and that requires VMware Tools,
and then, of course, the right version of VMware Tools.
So if we have a VM that we migrated over from 7, and when we got it to 8, we didn't upgrade VMware Tools, this may not work.
We want to make sure we have the newest version of VMware Tools.
If we go through and we update our version 8 ESXi to 8.0 update 1, update 2, update 3, whatever it may be,
and that comes with new versions of VMware Tools, and we don't update the VMware Tools in the VMs that are running this SR-IOV feature,
it may break that virtual machine until we upgrade VMware Tools over there.
So now, all of a sudden, we're making our patching a lot more critical,
and our network way more reliant on making sure when we patch the hosts, we update VMware Tools in the VMs.
I've seen it many times where I say, okay, we're going to be patching hosts this weekend,
and we're going to schedule it for later to patch the VMs because I can patch my hosts and not take anything down,
but to patch the VMware Tools or update the VMware Tools in the VMs requires a reboot,
and reboots of the VMs require a maintenance window, and that means scheduling.
That means a bunch of people have to be notified that these certain services this time go down.
Certain VMs have to go down in a certain order and then come back in a certain order in certain environments or certain applications.
Because of the complexity of patching certain VMs, a lot of VMware admins don't patch VMware Tools very often.
That is not a good practice.
You should patch your VMware Tools as often and as frequently as you patch your hosts,
but in the real world, that's not exactly how it works.
In the perfect world, it would be.
It's not always how it works.
So because of that, now this can be dependent.
Your network operation can be dependent upon having the correct version of VMware Tools in your VMs to match your hosts
so you can get the right version of the driver so that all of this works, okay?
So again, this is going to be dependent upon a lot of different pieces.
But if you have this working correctly, you can make your network performance be a lot faster,
take away a lot of overhead from your hosts, okay?
So if you want to know if you have the right host and the right network cards and all that kind of stuff,
check your VMware Compatibility Guide.
So here's the path.
Your guest OS requests a change for a virtual function, okay?
So the virtual function forwards the request to your physical function, the actual physical network card itself, right?
The physical network card driver checks for the request.
The virtual switch verifies, right?
So example request against whatever policies you have configured on the networking there, right?
And then your physical function, your actual physical device driver configures this virtual function, settings are compliant.
So there are policies matched with the virtual machines trying to do, all right?
How do we actually enable this?
Here's the important part.
So we need to go to our vSphere client.
We're going to go to our host.
On our host, we're going to go to configuration and networking and then go to the physical adapter.
We're going to select the adapter, and then we're going to go to the settings, and we're going to enable SRIOV.
If you right-click on your physical adapter and go to settings and you don't have any option there other than maybe like, you know,
it's set to 10 gig or 1 gig or whatever the case may be set to, that's the only option you have.
Then your SRIOV is not on that card.
You don't have an SRIOV compliant compatible card.
Or if you do, you don't have a driver for ESXi or ESXi call it a VIB.
You don't have the right VIB for that card to use that function.
So maybe, you know, the card's newer and the VIB's newer and you need to flip the VIB into the, you know, the ESXi host in order for it to see that card and see all the functions of that card and be able to use the SRIOV.
So if you right-click on, if you go into your, you know, your environment at work and you right-click on your adapter and go to settings and you're like, it's not there, your card just isn't supported for it.
Either this doesn't have it, which is probably the case, or if your card does support it, we wouldn't got brand new, the latest and greatest cards.
Are they on the hardware compatibility list?
Does VMware, you know, supply a driver with the operating system that match that?
Or do you need to slip the VIB in for that?
And in which case, is it certified by VMware to work with your host?
Okay, so some things to consider there when you go to edit that setting if you do or don't see this option.
So once we've enabled the SRIOV on the adapter, then we're going to set the number of virtual functions.
So we're going to set the number of queues.
So we can have multiple queues and different queues can have different options set up.
Maybe we want to enable certain features.
We'll talk about them.
Basically, all the features that you're going to look at on a port group, are we going to allow, you know, MAC spoofing?
Are we going to allow promiscuous mode?
Things like that.
For every group of settings you need for that, you're going to need a different virtual function, okay?
You're going to set those policies on the virtual function.
Are we going to allow, again, promiscuous mode, MAC address spoofing?
You know, things like that on there.
Say, okay, we're going to restart that host.
On the virtual machine, we're going to then go to the VM and we're going to want to add another NIC to the virtual machine.
And for the NIC, we're going to say, you know, instead of being on E1000 or VMXNet3, we're going to say we want an SRIOV network adapter for this particular virtual machine, right?
Then we can, you know, select the VM and we're going to specify what adapter we want to put on there, okay?
So, then you're going to assign a virtual function to that adapter.
We're going to add the adapter.
We're going to add the adapter for the, you know, SRIOV and then you'll have an option on the adapter for selecting the virtual function that we want to use.
And so then, again, we have in there what are all these different options we want to do.
Do we want to change the MTU size?
Do we want to enable jumbo frames on this one for some reason?
Are we going to allow promiscuous mode?
So, some of the other options that I mentioned, you're going to say okay to that and then you're going to power on the virtual machine.
So, we power on the virtual machine.
You're going to have this new network card and this new network card is going to be associated with this new virtual function that's going to be able to receive packets directly loaded into its memory from SRIOV-compliant physical adapters.
So, some of these options that we mentioned, MTU size, are we going to allow promiscuous mode on this adapter or not?
You know, basically any kind of traffic shaping you want to configure on there, so your security policies, that kind of stuff, right?
Also, your DLAN tagging.
So, keep in mind that when we get information from the network in this SRIOV NIT, we circumvented the virtual switch and the port group.
All of that information that the virtual switch and the port group had, like my DLAN tag, my MTU size, my security settings for MAC address spoofing and all that kind of stuff, promiscuous mode, all of that was circumvented when I circumvented the port group or the virtual switch and the virtual switch.
So, now I need to get those settings for all that stuff from the virtual function that I created on the physical NIT when I enabled SRIOV on it.
So, some different modes that we can use for these virtual functions.
Mixed means that I can support both SRIOV adapters and I can support ones that don't support SRIOV, right, or VMs that are supporting both, right?
SRIOV only means I'm only using, you know, physical adapters to support it and only communicating with VMs that support SRIOV.
Non-SRIOV mode means I am not using SRIOV on the adapter and I'm not sending any SRIOV traffic to any VMs that are using SRIOV.
So, that means that it's all going through the normal channel, from the adapter, through the kernel, through the port group, to the virtual NIC on the VM.
So, some common known issues, again, VMware tools.
If your VMware tools aren't installed, you're not going to be able to use the SRIOV adapter on that virtual machine.
If you don't have SRIOV enabled on a host, you won't even have that adapter as an option to install on a virtual machine.
So, make sure you have VMware tools installed and make sure that the current version for the version of EFXI you're running.
Check the compatibility list, right?
You don't want to use a device that's not on the hardware compatibility list.
You don't know how the kernel is going to respond to it.
You don't know if there's going to be bugs or issues.
VMware, when they release updates for the kernel, for the host, aren't going to worry about this non-signed driver working or not working from their update.
If they're releasing updates, they're going to try and write them in such a way that they're compatible and compliant with the rules they set for their vendors,
the ones that are using the correct APIs and the ones that are on their hardware compatibility list.
If some vendor is trying to sneak in and not pay the money to be on the hardware compatibility list to not be tested by VMware,
they're also not being tested to know if they wrote to VMware specifications.
And so, then when VMware runs updates, they're not worried about people who didn't write their specifications.
And so, their update might break your adapter.
This function that used to work but wasn't advertised, it wasn't a documented function that people knew was there, they can get rid of it.
They can change it, and it'll start breaking things because they didn't say,
this is a supported function that you can write to, things like that.
So, make sure your hardware is on the hardware compatibility list.
Resource management and limits.
So, you can run these systems, and you might want to enable logging.
You might want to enable monitoring of the health.
You want to make sure that you are keeping track of what's happening.
What is the utilization on the card?
Because, again, this card is doing a lot of work on its own.
We're not going to the CPU anymore because this card has, you know, it's a smart adapter.
It has a chip on there to do all this work and verify all these settings for me.
But it's not as big and huge a chip as the CPU that, you know, from Intel that's running the whole server.
So, it should be, you know, sufficient for running your workloads.
But if your workloads become heavy, and you have a lot of different tasks you've assigned it to do,
you've given it, you know, a larger number of VRFs, a lot of custom options, and all these different VFs.
Then, all of a sudden, that is going to increase the load on the card.
You might start seeing the card becoming overloaded and getting slower performance on there.
And then, of course, logging limitations.
So, logging is a resource-intensive operation, whether it be on the kernel, whether it be from a card.
Logging is a resource-intensive operation.
So, the more you log, the more detail you log, the more overhead logging is going to create.
Now, sometimes that overhead is absolutely necessary and worth it because you've got a problem you need to fix.
And you need that logging to fix that problem.
But once you fix the problem, once the logging is no longer needed,
lower the logging level to whatever is acceptable in your organization from being that max debugging.
So, maybe I just want information now.
I just want, if there's a problem, there's a warning, let me know, right?
Don't give me every little detail, though, for every event.
Just give me, you know, warnings, give me events, give me critical problems.
So, don't give me all that little detailed information.
Now, when there's a problem and we're, you know, trying to troubleshoot it, go to debug mode and give me everything.
And then, again, once I fix that problem, I'm going to lower it back down to only giving me warnings
and only giving me, like, critical events and things like that.
So, know what to log and when to log it.
Also, it's a good idea to send your logs off the server because sometimes it'll come up in the server, it'll crash,
and you're like, what caused this crash?
Well, the logs are on your crash server.
Go figure that one out.
But if we're dumping logs to a syslog server, maybe that log, that error got logged to the syslog server before the system crashed.
We can see what's happening and led up to the crash to help us understand what happened there.
So, logging limitations are great.
So, that is our single-route IO virtualization.
Very cool feature.
If you have a network card, it's going to be a more expensive, higher-end network card that supports the function,
and we have it on the hardware compatibility list from VMware,
it's actually a pretty cool thing that we can enable on our ESXi host now.