It's not going to be news to anybody taking this course that our network must
be online
as much as possible.
Once upon a time, the network was nice to have, but these days it's absolutely
required.
The reason it's required is because our users can't do their jobs without
having access
to the applications that allow them to do their jobs.
Yeah, maybe once upon a time again, we were able to do things with pencil and
paper,
but if I don't have access to my applications, then now I might as well just go WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
but if I don't have access to my applications, then now I might as well just go
home.
I'm reminded of the time that I worked for a medical clinic and from my
perspective,
being in the IT department, I knew that I had to keep the wireless network
online and one
day it was starting to go down.
We had this bizarre issue that we had to troubleshoot through and eventually we
found the issue
and we got it figured out, but it did take about an hour and a half, something
like that.
Well, it just so happened that my wife was going to the doctor that day at the WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
Well, it just so happened that my wife was going to the doctor that day at the
very medical
clinic at which I worked.
She had our young son with her and they actually experienced the outage from
the other side,
where doctors and nurses were coming in and apologizing to her because our
system is
down and we can't do anything, we can't access medical records and we can't
administer medications.
Sure, there are some things that the doctor and nurse could do in the meantime,
but for WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
but for
the most part they had to wait for the network to come online and they were
really close to
sending our home.
It's a reminder to us that even though we sit on the networking side and we
know we need
to keep the network online, that real lives are affected.
Not necessarily, like no lives were lost that day, not from that perspective,
but from the
perspective of the staff, the medical staff, they were embarrassed, they were
flustered, they
were frustrated, they were unable to do their jobs because the network went WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
were frustrated, they were unable to do their jobs because the network went
down.
This is why we need to keep the network online as much as possible because, hey
, I know that
I want to be able to do my job and if technology gets in the way of that, my
drawing tablet stops
working, for example, it can be very frustrating because I just want to do my
job.
I don't want to have to deal with all these technical issues.
So that's great, Null, except how do we go about doing it?
Well there are two primary things to consider. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
Well there are two primary things to consider.
The first aspect of this is going to be fault tolerance.
Fault tolerance asks the question, are there any single points of failure in
the network
or maybe more succinctly, can I survive a failure?
We need to look at all aspects of the network in order to figure out if we can
survive a failure.
And again, the goal is to identify those single points of failures. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
And again, the goal is to identify those single points of failures.
If we can identify the single points of failures, then we can eliminate those
and that's going
to help us to survive when a failure does occur.
It is important to throw out this entire conversation for us to recognize that
failures do occur.
We can't stop hardware from failing.
We can't stop maybe protocols intersecting in ways that they shouldn't.
I once actually took a pair of switches like this and we had a connection down
to an access WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
to an access
layer switch and I plugged this access layer switch into the other distribution
switch and
it completed a loop.
I plugged it and tried the exact same thing again and it worked just fine.
That's technology.
Sometimes it doesn't behave the way it's supposed to. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
Sometimes it doesn't behave the way it's supposed to.
So this is why we say we can't avoid failures. We just need to make sure that
we have the ability
to survive when a failure occurs.
So we need to look at a few different aspects of this, especially as it relates
to our network
topologies.
So what can we do to make sure we have no single points of failure?
Well, we need to look at the critical parts of our network and make sure that
we have
redundant hardware.
This is going to be a big part of this because, again, hardware can fail and if WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
This is going to be a big part of this because, again, hardware can fail and if
we lose a
switch, we need another switch available to take over for it.
So this is especially going to be true at the core block, the distribution, or
data center
aggregation block.
And hey, even at the data center access layer, we might need redundancy and we
'll talk
about data center coming up.
But we don't think about necessarily connecting a user lap top up to two
different access
layer switches, but in the data center, we'd absolutely want to connect our WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
layer switches, but in the data center, we'd absolutely want to connect our
servers up to
multiple switches in case one piece of hardware goes down.
We also want to look at appliances in our network.
So when we're talking about appliances, we're usually talking about firewalls.
We might be looking at load balancers, certainly wireless lane controllers.
If we have multiple wireless lane controllers, then we need, at that point,
hopefully we can
survive a controller going down.
But if we lose access to all of the wireless lane controllers in our WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
But if we lose access to all of the wireless lane controllers in our
environment, then our
lightweight wireless architecture is going to come to a screeching halt.
And that's not good for anybody.
So those are some ways that we can look at this.
We also want to look at the network connections because the network connections
are going to
define whether there are single points of failure as well because I can have
hardware
failure, but it doesn't do me any good if my hardware is connected like this.
And so there's a redundant pair. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
And so there's a redundant pair.
And yet there's only a path in this way.
So if this link goes down, I have no access to the redundant set.
So we need to make sure, for example, that maybe we've got a pair of switches
down here.
And so we want to make sure we have a full mesh of connectivity.
That's going to ensure that we maintain our online abilities and status if a
particular
piece of hardware or for that matter, a particular link were to go down.
We also want to consider the concept of port channels or the Cisco term for WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
We also want to consider the concept of port channels or the Cisco term for
that is
ether channels.
If we have redundant connections between two devices, so now I've got a device
here and
a device over here and we need to connect them at layer two.
Well, if I connect multiple connections like this, we're going to perform a
banding tree
blocking operation on one of those.
And if the one of those links goes down, it's going to create spanning tree TCN
s to flood
the network.
And so we're rather than allowing for us to run spanning tree here, we bundle WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
And so we're rather than allowing for us to run spanning tree here, we bundle
these connections
together and ensure that it's a single logical link.
That way if one of the connections goes down, it doesn't impact my layer to top
ology.
So that would be something to consider there as well.
From here we can look at different protocols that we should be establishing.
So if we're going to deploy redundancy at the protocol layer, what kinds of
things should
we be looking at?
Well, from a layer three perspective, we absolutely want to be running a WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
Well, from a layer three perspective, we absolutely want to be running a
dynamic routing
protocol.
So OSPF or EIGRP or even BGP in larger environments.
And that really is saying, hey, let's make sure that we're not running static
routes.
Even in a smaller environment, we might think that we could just deploy a few
static routes
here and there.
But if a static route, maybe even one that's being injected into a dynamic
protocol, if that
goes away because a link goes down, well, that could actually cause a failure.
So we want to rely on the dynamic nature of these routing protocols rather than WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So we want to rely on the dynamic nature of these routing protocols rather than
static
routes where it's possible to do so.
And coming off our conversation, if you're working your way through the course
sequentially,
we just talked a lot about gateway redundancy via FHRP's.
So that would be deploying HSRP, VRRP.
And in some cases, maybe GLBP.
Now from here, we want to look at other ways that we might experience a single WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
Now from here, we want to look at other ways that we might experience a single
point of
failure without maybe even realizing it.
As we just said for these connections, we want to make sure we have a full mesh
and we've
got port channels.
But think about this.
What happens if our pathing causes us problems?
So I might look at a logical topology that I design and say, all right, I'm
going to design
redundancy into this solution.
So I'm going to make sure that I have these two remote devices redundantly WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So I'm going to make sure that I have these two remote devices redundantly
connecting
to one another via a port channel.
But because these devices are in two different buildings, I'm going to rely on
fiber to connect
the switches.
So this looks like a redundant design from a logical perspective.
However, what is happening physically?
This is the question, especially when we're talking about some form of fiber
connection,
dark fiber, fiber that's in the ground that we're going to put optics to. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
dark fiber, fiber that's in the ground that we're going to put optics to.
So what happens if I have a physical building here and a physical building here
?
Let's say we're looking down at the building from the top.
So these are the roofs of the building.
And so I've got my Cisco device here and my Cisco device over here.
And again, logically, they're directly connected to each other.
But physically, one fiber connection comes out of the building here and comes
around and
goes, and okay, that's great.
But what if the other fiber connection uses the exact same physical path?
It wouldn't you know it, something comes by and cuts the conduit and breaks all WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
It wouldn't you know it, something comes by and cuts the conduit and breaks all
the fiber.
What happened to my redundant design?
It was just taken out with a single backhoe or crane or lawn mower.
We've all heard probably plenty of those stories.
They're all unfortunately very real and very true.
So instead, we need to make sure that this other fiber connection takes a
different physical
path.
So that if a backhoe comes by and digs up the fiber by mistake, and then at
that point
we still do have our redundant connection. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
we still do have our redundant connection.
So we do need to make sure that our path is secure when we have control over
our own network.
So this would be more of a local area network type of conversation.
But then when we're looking at a wide area network conversation, especially
with a, well,
say an internet service provider, but it could just be a normal YN service
provider.
At that point, we need to ask the question of what about the service provider
network?
Because again, I could design a beautiful redundant solution and say that I'm WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
Because again, I could design a beautiful redundant solution and say that I'm
going to have my
two edge devices communicating with two devices on the service provider.
We're going to have a full mesh.
And we've, you know, if any of my hardware goes down or your hardware goes down
, we maintain
uptime.
And that's great.
But what if both of the service provider routers connect to the same upstream
device and
it fails?
Unfortunately, I've heard stories about this too.
And so this kind of thing can actually cause that this beautiful redundant WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
And so this kind of thing can actually cause that this beautiful redundant
design to have
a single point of failure.
So usually because this is outside of our control, we need to at least be
asking the questions
and maybe even asking for some form of topology map that can prove that the
service provider
is giving me a redundant design.
Now other than that, from a fault tolerance perspective, we can look at
hardware components.
For example, redundant supervisors, redundant power supplies, power supply
units here. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
units here.
And if we have redundant power supply units, then we need to make sure that we
have different
circuits.
So circuit A and circuit B, because if they're both plugged into the same
circuit and that
circuit pops, well, that we just boss both of our PSU.
So same kind of thing here where we think we have redundancy, but we really don
't.
So lots of considerations that we should be making on the fault tolerance side.
Now the second thing we need to consider, yes, that was all number one.
Now we're on to number two.
Number two here would be high availability. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
Number two here would be high availability.
And high availability conversation is actually one that is meant to find a
balance between
the amount of fault tolerance that we're going to deploy and the amount of
money that we're
going to spend.
And this ultimately, in most cases, is going to be a management decision, but
they can't
make that decision if we don't provide the amount of data.
So what we're saying here is essentially, how up are we going to be, or maybe WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So what we're saying here is essentially, how up are we going to be, or maybe
we should
say, how much downtime are we going to allow down downtown, down time?
There we go.
How much downtime will we allow?
And again, this should be a management decision.
Now every manager, at least if, unless they have an ID background, they're
going to come back
and say we want to be 100% uptime or 100% up.
And okay, that's great.
I just need $14 million in order to ensure 100% uptime because the reality is WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
I just need $14 million in order to ensure 100% uptime because the reality is
that we can't
ever guarantee 100% uptime.
We said it over here that failures are going to happen.
And even in the most robust, securely designed networks, failures will occur.
And we hear about those failures pretty commonly from major vendors who you'd
think would
have a very robust network design and they probably did.
So it's not about 100%. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So it's not about 100%.
It's usually about the nine's conversation.
How many nines of uptime do we want?
So for example, if I say I want a 99% uptime, this would be two nines of uptime
because there's
two nines in the number of 99.
Or if I can say, let's go to 99.9% uptime, well now that's three nines of up
time and so on
and so forth.
So maybe we'll say 99.99% uptime.
And we might even be able to get a conversation going where we have five nines. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
And we might even be able to get a conversation going where we have five nines.
Six nines is, yeah, you're really getting up there at that point.
Now anytime we're talking about a percentage, we can apply it back to the
strictly speaking
the number of hours and minutes we have in the year.
So we have a random calculation here, 8,760 hours and 525,600 minutes per year. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
So we have a random calculation here, 8,760 hours and 525,600 minutes per year.
So if I'm going to say 99% uptime, well I can look at my hours there and take
one percent
of that basically and say that we can be down for 87.6 hours a year.
Holy moly, that's more than three days.
99% sounds great, but it's really low bar.
We probably want to do a little better than that.
Now if we advance to 99.9, we can divide that number by 10, essentially. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
Now if we advance to 99.9, we can divide that number by 10, essentially.
And so we're talking about still 8.76 hours.
And if we get to 49, at that point we're below an hour, 0.876 hours that
translates to 52.
We look at this number over here, we can round that up to 0.6 minutes.
And a lot of networks aim for 49.
It's a pretty good balance of cost versus acceptable outages.
And if we want to go to 59, then we're at 5.26 minutes. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
And if we want to go to 59, then we're at 5.26 minutes.
And if we go to 6.9s at that point, we are about a half a minute, roughly 31.5
seconds
of outages per year.
Now the problem with this is that as we go up the scale or down the scale, how
we want
to say this, it's going to require more money.
We need higher resiliency switches with redundant power supplies, redundant
circuits, now
become redundant power grids, redundant service provider connections become WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
become redundant power grids, redundant service provider connections become
redundant path
connections.
We need to maybe dig more fiber and deploy more complex protocols.
It might require additional staff and better monitoring tools.
We have a lot of needs if we're going to talk about that level of high
availability.
And unfortunately limitless budgets don't exist.
So fortunately at the end of the day, as IT engineers, this is still a
management decision.
But we have to provide them with all of the information required in order to WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
But we have to provide them with all of the information required in order to
make that
decision.
And then if they decide, hey, you know what?
Yeah, we want 100% uptime, but we're only going to give you budget for 3.9s of
uptime.
Well, okay, at least we've spoken our piece.
We've communicated that we can still expect up to 8 hours, maybe even up to 9
hours of
downtime a year at this budget level.
And they're just going to have to deal with that when the time comes.
But at least at that point, we have done our due diligence. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000
But at least at that point, we have done our due diligence.
We have provided all the redundancy that we can.
We have considered all of the single points of failure in the network.
And we have designed as robust of a network as we can given the existing budget
ary constraints.