[nycbug-talk] breaking up a big cisco

[alex at pilosoft.com]

On Tue, 11 Apr 2006, N.J. Thomas wrote:

> Will be expanding in the near future, adding another Tier1, as well as
> anticipating 50-100% growth in the number of servers in about 2-3 years
> time. The 6506 being so large and expensive is not a major issue in and
> of itself, but because it is a SPOF, its size and cost does affect our
> plans.
> 
> So the proposal was to split the whole thing up. Roughly, the
> architecture would be:
> 
>     - arrange servers into 4 cabinets
> 
>     - put a switch in each cabinet (Cisco 2960?), to which each server
>       would be connected
> 
>     - have the carriers go into a switch which feeds into a smaller
>       router (Cisco 2821?) which would go out to the switches which the
>       servers are connected to
> 
>     - have spares available on hand for all switches and routers in case
>       something fails
> 
> Our proposed network would probably look roughly something like this:
> 
>     [Tier1] [Tier1] [Tier1]
>        |      |       |
>        +-+    +    +--+
>          |    |    |
>       +--+----+----+--+   +--+----+----+--+
>       |  Cisco  2960  |   |  Cisco  2960  |
>       +-------+-------+   +-------+-------+ --- servers
>               |                   |
>       +-------+-------+   +--+----+----+--+
>       |  Cisco  2821  |---|  Cisco  2960  | --- servers
>       +---------------+   +-------+-------+
>                                   |
>                           +--+----+----+--+
>                           |  Cisco  2960  | --- servers
>                           +-------+-------+
> 
> Is this a viable solution? How would you modify it to provide some
> measure of redundancy?
> 
> Another thing I am worried about is the whether or not the 2821 router
> can handle 3 carriers. Our traffic needs are fairly low, our Tier 1
> carriers are both 5Mbit burstable, and we rarely go above 15Mbit during
> peak times. We'd be adding another carrier only for redundancy, not for
> the bandwidth. However, because we are an ASN running BGP, I'm not sure
> if the 2821 router would be constrained by memory limitations.
1) find a network guy who knows what they are doing. 

You omitted crucial things: 

* what supervisor do you have on 6506. If you have sup1, yes, it may time
to throw that junk away, not much of reusability here (but even then, if
you are doing 15Mbps, sup1 will handle just fine). If you have sup2 (or
sup720), it is *great*. Don't even think about 2821s if you already got
6509 with sup2. Buy another one for redundancy, 6506 with sup2 isn't all
that spensive. 65xx as a platform is few orders of magnitude more scalable 
than 28xx.

* do you have a 'flat' network or do you have servers segmented into
vlans, and different IP blocks on each vlan?

a) The *preferred* way is to do layer 3 on each of your 'distribution'
switches (the ones in each rack), and do OSPF between each of the
distribution switches and your "core" (whether the core is 2821 or 65xx).
2960 is not a layer 3 switch. If you go this way, get 3560 or 3570.

Then, each of the 3560s connects to each of the 65xx (or 2821), and
announces reachability with OSPF.

b) If you want to do layer 2 on distribution switches (your 2960s), that
is fine. Failover in this case will be accomplished by STP/RSTP. STP can
*really* mess your network up if you don't know exactly what you are
doing. Again, in this case, each of the distribution switches is connected
to each of the core switches. In this case, you will be running VRRP or
HSRP on the core switches to provide for failover.  This config is more
"fragile" than the configuration a) above.

The physical config in both cases will look the same. The logical (IP) 
will be very different.

* redundancy among upstreams is accomplished by putting some upstreams on 
one core switch, and other upstreams on another. 

* 2821 can do 15mbps just fine, and can handle full tables fine.