It's going to be very common in STWAN that we have multiple paths to a subnet.
For example, here in our lab environment we have edges 11 and 12 at different locations
and 13 and 14 are part of the same site and they're both going to want to advertise subnet
A. 13, for example, is going to advertise it as being connected to its internet T-lock
here, so 11 and 12 know they can get to it this way, but 14 is also going to advertise
it out its internet T-lock and so they know they can go this way.
But we're not done yet because 13 is also going to advertise it out the MPLS T-lock,
which means that 11 and 12 can get to it this way and then 14 is going to do the same.
So when I look at it from edge 11's perspective, I have four different paths that I can go
in order to get to subnet A. I can use the internet pipe to get to either 13 or 14, and
for that matter I can use the MPLS pipe to also get to 13 and 14.
So as we dive deeper now into this world of routing in the SD-WAN space, and specifically
when we're looking at things from an OMP perspective, we're going to need to understand how this
concept of path selection works.
This is going to be approached from a couple of different angles here, not the least of
which is the ability for us as admins to be able to perform some amount of traffic
engineering.
Because if I look at this at face value and I say that edge 11, again over here, has four
different paths, these paths by default are all going to be seen as equal.
And so do I want my routing table here for edge 11 to list that subnet, which in our
lab environment is 10.10.13.0.24, and do I want it to see four different equal cost paths?
If so, it's going to perform load balancing, and I'm going to send one flow this way, and
one flow this way, and another flow this way, and another flow the other way, and it starts
to get at least a little bit of, not chaos per se, but we're going to be sending our
And one of the advantages that SD-WAN is supposed to give us is as much control as possible
over which traffic is going out which pipes.
Now an important part of this conversation is to remember that edge 13 is not actually
advertising anything to edge 11.
Instead we have our SD-WAN controllers down here, otherwise known as the vSmart, and all
of these edges are performing advertisement down here to the controller.
And then the controller reflects those routes back out.
And so what we're really talking about happening here is the SD-WAN controller is going to
aggregate a bunch of routes, and so for example, 10.10.13.0.24 will be one of them, and then
it's going to recognize that there's, again, four different paths here, and then send those
back out, which is why edge 11, as our stars accumulate over here, this is why edge 11
knows about the four paths.
It's not because edges 13 and 14 told edge 11 anything, it's because they told the controller
this way, and then the controller told edge 11 by sending it back out.
So there's an inbound component to routes being advertised, and there's an outbound
component.
We need to start thinking about it in these terms, because as we start to deploy policy
against this, we're going to recognize that we can do certain things to routes as they
come into the controller, and then also certain things to the routes as they go outbound from
the controller.
So what happens here is, once again, we're going to aggregate all of the paths to a particular
subnet, and then we're going to advertise the best.
The only advertising that we do is with the best path.
Now the concept of best is what we're going to be spending some time on.
How do we define what the best path is in SD-WAN?
We don't have a lot of the same concepts that we use in EIGRP and OSPF, and even BGP doesn't
really fit this specific scenario, even though we've said many times that OMP parallels BGP.
So we need a new way to define which route is best, which as we might imagine, is going
to impact this conversation with traffic engineering, because if I have different parameters I have
to adjust, well I need to know what those parameters are, and OMP fortunately defines
that for us.
But for now, let's take for granted that OMP is able to determine what the best option
is.
So it's going to aggregate these, let's say four routes, and it's going to say this is
the best one, and send it out.
Now in our example we said that they're all equal, and by default they certainly will
be.
So when we say the best, this could be one, or it could be many paths, and this is the
conversation that we have in the routing space known as equal cost multi-pathing, or ECMP.
The idea here is that if I have routes that are tied for best, that I should use all of
them, and I should load balance across them, which is what we set up here.
We will load balance by flow across all of our different paths going to a specific, following
a specific route.
So once again, if there's one best route, it's going to advertise it.
But what if we have more T-locks on these devices, and maybe we have three T-locks on
each router, and now we actually have six paths?
Are we still going to continue to advertise all of these?
And when do we cut it off?
Am I going to send 20 best paths?
And the answer is, well it depends.
It depends on a configuration parameter known as the send path limit.
The send path limit is a configuration that goes on the controller, and it tells us how
many routes that we're allowed to send outbound, down to the routers.
By default, we're only going to send four, which means that the SD-WAN controller has
to number one, aggregate these routes, but number two, it has to order them.
These must be ordered, and so we'll see how exactly we go about ordering them, but the
idea here is we're only allowed to send four by default, and so we'll send the best four
if these are all equal.
I want to continue to emphasize that.
Only the best is advertised.
The only time we send more routes is if we have a tie, so if these are all tied, then
we will still only send the four paths, these top four paths.
We won't send these bottom two paths even though they are technically tied, so the question
is how do we order something that's tied?
Well, we're going to need some arbitrary tie breakers, and so that's what again we'll be
talking about coming up here real soon.
If I wanted to, I could set this all the way down to one, or I could increase it to 32,
so depending on my environment and how many paths I want to be taking and what is optimized
from a load balancing perspective, I might want to increase this.
I might want to send all six of these equal cost paths, and again, we can expect to see
a lot of equal cost paths in this environment because these T-locks are all going to look
equally appealing, especially from the same device like Edge 13 there.
All three of these T-locks will look pretty much the same until we start to get involved
from a traffic engineering perspective, and again, once Edge 14 starts advertising as
well, then we're probably going to see a bunch of these equal cost paths.
Let's say that we do set it to six, so we decide that we are going to set it to six
equal cost paths using the send path limit, and so how exactly are we going to expect
that the Edges will receive these?
Well, again, I'm going to advertise this, so let's say I advertise this to Edge 12 here,
so I'm sending six equal cost paths.
Well, by default, 12 is only going to install four of them, and why is it only going to
install four?
Well, it's going to install based on a parameter that's set on the Edge, which is known as
the ECMP limit, so we have two different limits happening here.
This is why it's so important to recognize these different directions and understand
how these routes are flowing in the environment, so the controller wants to receive all of
the routes that are being advertised.
We want to aggregate everything together.
limit, and again, I will continue to stress that the send path limit only affects equal
cost paths.
If I have five routes and they're all different, I'm not going to send four of those.
I'm only going to send one of them because one of them is the best, but if we're talking
about equal cost paths, I'm only going to send a default of four, but I can configure
it to send more if I so desire.
I go to all the trouble of configuring it as six paths, and now we've got another limit
here, the ECMP limit, which also has a default of four.
It would make sense that they'd match by default, otherwise we'd be sending routes that we can't
receive without ever changing the configuration.
That would be a little bit inefficient.
If I'm going to increase this to, let's say, six, then I probably need to increase this
limit to six as well, so that it will install six equal cost paths because that's how many
I'm advertising.
We need to be aware of these two limits and understand that one is affecting how many
I'm sending, and then one is affecting how many I'm willing to install.
This configuration, let's be clear, this configuration's on the edges, and then the send path limit,
I guess I already have the error over here, but this is a controller, an SD-WAN controller
or a vSmart, however you want to call it, but it is a controller configuration.
All the way back to this concept right here.
How exactly am I ordering these routes?
How do I decide, number one, which is best?
That's a very important question.
For that matter, we mentioned some tiebreakers because if by default all routes are going
to appear the same, then we're going to need some level of tiebreaker in order to create
an actual ordered list.
Otherwise, we just have a bunch of tied routes and we don't know which ones should be preferred
over the other, which as we can tell here, might end up mattering, especially where these
limits are concerned.
The answer for all of this is that we're going to have path selection criteria as part of
OMP.
It's a similar concept to how BGP operates, where we're not using the concept of metric
like EIGRP or cost like OSPF, we're not 100% focused on bandwidth because it's not just
a bandwidth or delay conversation.
There's got to be other factors taken into account with something like OMP.
This list of criteria is going to determine which routes are better than others, and that
will allow us to aggregate a bunch of routes and then create an ordered list.