All right, so for the next flow,
from VM-1 to VM-3 is from here going over here.
So from a routing perspective,
it's as if these two VMs
live on the same logical layer three network which they do
and they're also connected
to the same layer two broadcast domain,
this logical overlay segment, segment 10.
And as a result,
there shouldn't be any routing decision whatsoever.
However what will happen is that when this client
tries to communicate with this device,
if it doesn't have the layer two address or VM-3 it, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
if it doesn't have the layer two address or VM-3 it,
it'll do an ARP request,
and this host can actually respond on behalf of this VM
and not bother sending
the actual ARP broadcast request over the network
'cause if it did, it'd have to go ahead
and use the transport network, and send it across,
and that'd be received here,
and then deencapsulated just for the ARP request.
So if the ARP information is already known,
we can save that step by having ESXi-A
just respond back to the client with the ARP response
and never have to send
the initial request across the network.
And then after VM-1 WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
And then after VM-1
has the destination layer two address of VM-3
and tries to send traffic to that layer two address,
ESXi-A is gonna take a look
at that destination layer two address
and say okay, where does that live?
And then based on the tables that NSX provides,
ESXi-A would know that oh, that destination Mac address
of VM-3 is reachable through ESXi-B.
And fortunately we have a tunnel endpoint at both ends
and so at that point,
ESXi-A would just encapsulate the data in the Geneve tunnel,
ship it over to ESXi-B who would deencapsulate it, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
ship it over to ESXi-B who would deencapsulate it,
and then forward it onto its destination.
So for VM-1 to VM-3,
what we're predicting here is no L3 routing.
So if we do a Traceflow,
it should not even involve this gateway at all.
But what we should see is we should see some tunnels
being used for the Geneve tunnel as we encapsulate
and send the traffic from ESXi-A where VM-1 lives
over to ESXi-B where VM-3 lives.
So let's verify that next.
So back to Plan and Troubleshoot,
once again back to Traffic Analysis,
and I'm gonna go ahead
and collapse this to the left a little bit WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
and collapse this to the left a little bit
and we'll say a New Trace.
And click on Proceed
and for this new trace,
we wanna trace from VM-1 again.
But this time we wanna trace over to VM-3.
So we'll select VM-3
and there are the details
including the layer three addresses
and the MAC addresses associated with those two VMs,
and we'll click on Trace.
All right, so here it's showing us the logical flow
from this VM going through segment 10 over to this VM
and if we click on this little icon
for the host over here on the left,
it's showing us that this is ESXi-A. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
it's showing us that this is ESXi-A.
And then over here on the right,
this one should be ESXi-B,
the other end of the tunnel.
And that traffic logically just went through
this layer two broadcast domain
that we know and love as the overlay segment 10.
So to see the play-by-play,
we can go ahead and scroll up here
and take a look at the details.
So that traffic was injected into the network
from the position where VM-1's network adapter is.
It went logically through a distributed firewall
past those rules and then it went to the physical network,
and check this out, this is fantastic. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
and check this out, this is fantastic.
ESXi-A, as that traffic
was going over the transport network,
it was sent from the source IP address
of the tunnel endpoint for ESXi-A to the remote endpoint,
and that's the tunnel endpoint for ESXi-B.
Then ESXi-B received that on its local 22, et cetera,
and there's the remote side that it came from.
If we scroll down,
then it also went through a logical distributed firewall
and one of the rules that was there
that's permitting the traffic by default
before it was delivered to that client.
So if we looked at this MAC address for a moment,
this Mac address right there, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
this Mac address right there,
that's the Mac address of VM-3.
One of the things that ESXi-A had to do was to identify
okay, I know the traffic needs to go to this same segment,
but where in the world is that MAC address?
How did the host transport node ESXi-A
know that the MAC address ending in da:1d
was a MAC address on that same segment,
but reachable over through ESXi-B as in boy?
And the answer is NSX provided that information.
So what I also recommend we do real quick
is let's just validate that as well. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
is let's just validate that as well.
So I just made a little note that
that is the MAC address ending in da:1d
that's associated with VM-3
and in our topology,
that's this little VM right here.
Again, that MAC address ends in da:1d.
So let's head over to ESXi-A
and then in the NSX CLI interface,
let's verify that ESXi learned via NSX
that that layer two address was reachable
via the tunnel over to ESXi-B.
So here at the NSX CLI for ESXi-A, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So here at the NSX CLI for ESXi-A,
let's do a get,
and we can either type in logical switches
or we can type in segments just to get a listing
of our layer two broadcast domains that are set up.
So I'll use the more current option of get segments.
So if we're looking for the segment 10 yellow,
that's this UUID or we could also just use
the VNI number for that,
the virtual network identifier.
So let's do a get segment space.
I'll plug in that VNI number.
I'll copy and paste that space and a question mark
and let's ask for
the MAC address table associated with that. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
the MAC address table associated with that.
So if we type in a MAC table and tab it out,
this is the MAC address table associated with that segment.
So notice right here that we have
the MAC address ending in da:1d
and that's the MAC address associated with VM-3.
And here's showing effectively
that to reach that layer two address, it's not local,
but we'd have to send it over the tunnel,
to this tunnel endpoint of 22 et cetera
which is the tunnel endpoint for ESXi-B.
And this is the layer two address
associated with the other end of that tunnel.
So if we looked at the VM kernel adapter on ESXi-B WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So if we looked at the VM kernel adapter on ESXi-B
that has this IP address assigned to it, 22.22.22.22,
its layer two address associated with that tunnel endpoint
would be this one ending in 90.
And we can verify that right here in the vSphere client
simply by going to Host and Clusters,
going to ESXi-B, going to Configure,
and then going down to VMkernel adapters.
And here's that IP address, 22.22.22,
associated with VMkernel adapter.
If we expand that,
there's the MAC address ending in 70:90.
So again, ESXi-A has that all cached up, ready to go. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So again, ESXi-A has that all cached up, ready to go.
So if it ever needs to forward a frame
to the destination address,
it knows exactly who the other end point of the tunnel is
and it also knows the MAC address
associated with that endpoint.
It encapsulates, it puts on the right VNI, and ships it.
And for that same segment,
we could also go ahead
and take a look at all the tunnel endpoints as well.
So get segment and this is the segment ID
for our segment 10.
Then we can followed up with vtep-table.
That's the virtual tunnel endpoint table
and effectively that'll be a list of tunnel endpoints WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
and effectively that'll be a list of tunnel endpoints
that'll have some association, or connection,
or responsibility as far as that layer two segment goes.
So we'll type in vtep-table
as the option there and press Enter.
So here's our local control plane.
Our local VTEP is 11.11.11, et cetera.
There's our MAC address associated with our tunnel endpoint
and then these three right here
represent the tunnel endpoints for other nodes.
So 22 et cetera is ESXi too.
Should we need to forward traffic to one of our edge nodes,
we have edge one at 33, et cetera.
Edge two is at 44
and as we start to integrate a tier zero gateway WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
and as we start to integrate a tier zero gateway
and add gateway services such as NAT and VPNs and so forth,
we are gonna need to leverage the services of those VMs.
We'll start using those teps as well.
So going back to this one right here,
which is the tunnel endpoint, ESXi-B.
Once again, there's the MAC address
associated with that tunnel endpoint.
And in the event VM-1 was doing an initial ARP request
for VM-3's MAC address on the same segment,
if we hit the up arrow key
and we said instead of showing us the vtep-table,
let's take a look at the ARP table. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
let's take a look at the ARP table.
We'll type in ARP table, press Enter,
and this is showing us the ARP table
that ESXi-A knows about.
So if VM-1 which is at 10.10.11,
if it didn't ARP request
for the layer two address associated with 10.10.10,
ESXi-A locally can just respond back to it
and say here it is,
and the client wouldn't even know the difference.
It simply made an ARP request,
it got the response
and at the same time,
ESXi-A did not have to ship that broadcast,
that ARP request over the tunnel WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
that ARP request over the tunnel
and flood it effectively to any other transport node.
Because they had it cached locally,
it could feed the response directly back to the client.
So for the next flow,
what if VM-1 right here at 10.10.11
was gonna send traffic over here to VM-2
which is on a different host
and it's on a different subnet?
So the rule about routing in the world of NSX
is we're gonna route as close to the source as possible
and what that means is that ESXi-A
is gonna do all the logical routing.
So as the traffic is sourced from VM-1 WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So as the traffic is sourced from VM-1
and it's going to VM-4,
I'll circle those for reference,
logically the traffic is gonna go over the segment.
It's gonna come into the router on this interface,
be routed out, go over this segment,
and then forwarded down to this VM.
It's gonna be ESXi-A
that logically is making all the routing decisions.
So as the traffic comes into ESXi-A,
it's gonna look at the destination address,
consider its distributed router.
It'll make a routing decision
saying oh, this needs to be forwarded
to the 10.20.0 network. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
to the 10.20.0 network.
It'll choose the appropriate egress or outbound interface
and then at that point,
it'll consider the destination layer two address
which is on segment 20,
but it'll also realize that oh, that layer two address
that I'm trying to forward to is reachable
over the transport network
and it would know exactly who the other end point
of the tunnel is.
So it'd go ahead and use the Geneve tunnel,
encapsulate that traffic, forward it over to ESXi-B
who then deencapsulate it and forward it.
So when the traffic comes in to ESXi-B,
the routing decision has already been done. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
the routing decision has already been done.
So in the Geneve header when it specifies the VNI,
it's gonna have the VNI information for segment 20.
And then all ESXi-B has to do
is simply when it receives it, deencapsulate it,
and forward it to the VM that has that MAC address.
So to help reinforce that,
we're gonna once again use Traceflow to do that.
But I'd also like to capture that traffic on the network
so you can actually see over the transport network
right here being encapsulated as part of the Geneve tunnel.
I'd like to demonstrate that before ESXi-A forwards it,
it already put the segment 20 VNI number on there WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
it already put the segment 20 VNI number on there
because it already made that logical routing decision.
So let's do all of that right now
with the trace from VM-1 over to VM-4.
So back in the NSX manager,
we'll say we wanna do a New Trace, click on Proceed,
and then we'll specify that our source VM
is going to be VM-1.
And there's its IP address and MAC address
and then the destination is gonna be
VM number four at 10.20.0.21.
So there's its IP address, there's its MAC address,
and let's click on Trace.
Now meanwhile while this is going,
I'm actually capturing the traffic in the background WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
I'm actually capturing the traffic in the background
over the transport network.
So when that traffic goes over
whatever the VLAN is they're using for that,
that's gonna be copied to my Wireshark
and then we can take a look at the actual packets involved.
So here in the output, let me bring this full screen,
here is the logical representation of what it just did.
So the traffic was sourced from VM-1,
went through this logical segment, segment 10,
went to a router interface
that was connected to that segment.
It made a routing decision,
forwarded it down through segment 20 down to the final VM.
So on the left-hand side, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So on the left-hand side,
if we look on the host icon here, that was ESXi-A.
And up here, if we click on the icon here for the host,
that's also ESXi-A.
And what may be a little surprising
is that because ESXi-A was the closest to the source
and it had a distributed routing table,
if we look over here at segment 20 and click here,
check it out, it's also ESXi-A
because all that logic between the routing
between subnet 10.10 and subnet 10.20 was done by ESXi-A.
But then when that traffic WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
needed to go to the final destination,
at this point right here in our journey,
it was forwarded over the tunnel
and that's where ESXi-B received it
and was able to finally forward it to its client
or in this case to its VM, VM number four.
So I go ahead and close that full screen
and we'll look at the play-by-play to verify that.
So here the traffic was injected
in the network on behalf of VM-1.
It went logically through a distributed firewall
which allowed it to pass.
Logically it went through segment 10,
then it went to the router on the inbound interface,
and then it was routed out. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
and then it was routed out.
The outbound interface again, still being done by ESXi-A.
Forward logically over the pink segment.
Then at this point right here,
it forwarded over the Geneve tunnel
over the transport network
and that once again, is because of the tables
that are being maintained that let ESXi know
that hey, based on this destination layer two address,
you need to forward it over to this endpoint
so it can go ahead and continue forwarding.
And then the next one is ESXi-B receiving it over the tunnel
and then processing it
logically through the distributed firewall.
It passes by default because the default rules permit it WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
It passes by default because the default rules permit it
and then finally being delivered to the client.
So if we were to capture the traffic right here
on the transport network
and from host A's perspective,
it's using Uplink two and VLAN 101
and on my transport network
I have routing and everything set up.
So it's being delivered over here to ESXi-B
at its tep of 22.22.22, et cetera,
and it's using a different VLAN
for the support of that subnet where the tep is.
And again, my transport network
is forwarding on routing all that traffic.
So the traffic from the client
would've been sourced in segment 10. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
would've been sourced in segment 10.
So ESXi would make your routing decision.
Just logically like this router would that say
oh, it needs to be routed out this interface.
So that decision is made by ESXi-A
and then as a result of needing to forward it
to that final layer two address
based on that layer two address
being reachable over the tunnel
and then that traffic would be encapsulated
and forwarded over to the other end point.
And as far as the actual VNI
that was included in the Geneve header,
it would be the VNI that segment 20 is using
because once again, the routing was already done WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
because once again, the routing was already done
in the mind of ESXi-A
and now we're just delivering it
by forwarding it through the tunnel to ESXi-B
where it can deencapsulate
and forward it to the final VM.
Also before we jump into the capture,
let's just verify what the segment 20 VNI is.
So here on the vSphere client,
let's go to the Network tab.
It's gonna be super easy to see right here.
So there is segment 20
and if we go to Summary,
right here it's showing us the segment number of 73728.
So I'm gonna make a note that the Segment 20 Pink
has the VNI number 73728. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
has the VNI number 73728.
All right, fantastic, with that in mind,
let's take a look at our capture.
All right, so here is our capture.
So I had this being delivered
over to my management computer.
So the first four rows
are all about just taking the traffic that was captured
and shipping it over with GRE over to my workstation.
So the real story as far as the captured traffic
off the transport network that we care about is right here.
So as far as the IPS involved
for the transport network it was coming from,
this packet was coming from 11.11.11 WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
this packet was coming from 11.11.11
which is the IP address
for the tunnel endpoint on ESXi-A
going to the tunnel endpoint on ESXi-B,
and the ethernet header looks interesting to me.
Let's take a look at that for a moment.
So the destination is this MAC address, 70:90.
I recognize that as the literal layer two address
associated with the VM criminal adapter.
But what I'm actually a little surprised about
is this source Mac address right here.
The source MAC address based on Wireshark
and what it believes the identifier here,
it thinks it's an address owned by Cisco. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
it thinks it's an address owned by Cisco.
And what just occurred to me
is that that's because that traffic was being forwarded
by my transport network layer three routed infrastructure.
So on my transport network,
I've got a router function here
and I captured this traffic right here on the network.
So this would be from one of my router interfaces
which is a Cisco interface on VLAN 10
that's going to the tep, tunnel endpoint address,
of 22, et cetera that ESXi-B is using.
And that would explain why the source layer two address WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
And that would explain why the source layer two address
represents a layer two address on my router interface
that was making that forwarding
on behalf of the transport network
and the destination layer two address,
it was the actual destination address
of our VM kernel adapter 22.22.22.22,
which is acting as our tunnel endpoint.
So it's always fun to look at the packet captures
to help reinforce what we think is happening
and verify that it literally is.
So once that traffic was then delivered
to this destination IP address of 22.22.22.
At layer four it's using UDP, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
At layer four it's using UDP,
so here it has a VNI number.
Now this is in hexadecimal,
so thank goodness it's doing the conversion
for us right here
and confirming for us that the VNI number
in decimal is 73728
which is what we verified is the VNI for segment 20.
So this header is just representing
that this is for VNI 73728
and then once ESXi-B deencapsulates that,
here is the original content.
So this destination MAC address right here WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So this destination MAC address right here
is the destination MAC address of VM-4.
And we can see that right here
in the results of our Traceflow.
So there's the MAC address, c9:d3.
And then for the source Mac address,
the source MAC address of VM-1
which ends in d9:97,
that isn't the MAC address that we have showing up
inside of our capture.
We have some kind of a VMware address
and the reason for that is that this logical router
made a logical routing decision.
So ESXi-A, when it made that routing decision
just as a real layer of three router,
would it used its own layer two MAC address WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
would it used its own layer two MAC address
associated with this egress outbound interface
before making the forwarding decision
down towards the destination.
So for that virtual source MAC address
that we saw in the capture,
that'd be the source virtual MAC address
that this logical gateway is using
for its interface that's connected down to our segment 20.
And just for a moment 'cause I'm curious,
I wanna verify what that Mac address is.
So back at ESXi-A who made that routing decision
and has a copy of the distributed router running on it,
then we'll do a get gateway
and we'll plug in that virtual distributed router UUID, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
and we'll plug in that virtual distributed router UUID,
a space, a question mark,
and then we'll type in interfaces and press Enter
and there are interfaces.
And so the egress or outbound interface
would've been this one right here
and this is the MAC address that would've been seen
for the source on that frame as it was being sent
and being routed out of this logical router interface.
So again, that's one ending in 4452.
Let's take a look at the packet capture
and verify that matches up.
And sure enough, right there for the source Mac address, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
And sure enough, right there for the source Mac address,
the one ending in 4452
which is the logical interface on the logical router
that was making the routing decision
for forwarding that packet over down to the segment 20.
Then we have the layer three header for the original packet
which is sourced from the VM-1's IP address
going to VM four's IP address
and then the actual payload
which was an ICMP Echo request,
and that's what they used for the actual traffic
by default in Traceflow.
So let's do a quick recap.
We created this logical distributed router. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
We created this logical distributed router.
We connected it to segments.
We gave it the IP address we wanted to use
of .1 on each of the segments.
We then had our VMs on those respective segments
use that IP address as their default gateway.
I provided that via DHCP
or we could have done it
manually as well on the VMs, either way.
And then for the forwarding,
if both VMs are on the same host,
each host with the distributed router
can make those routing decisions
and forward the traffic without an ounce of that
having to go over the transport network.
Or when traffic does need to go,
for example, from a VM over here WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
for example, from a VM over here
on ESXi-A on that transport node
and be sent over here to one of these VMs
over here on a second transport node,
the transport node that is closest to the source
will make any routing decisions,
layer three routing decisions that are needed.
And then once those routing decisions are made,
if it needs to forward the traffic over the tunnel
to another transport node,
it'll do exactly that.
And then it's up
to the receiving transport node to deencapsulate
and forward to the correct and final destination.