Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
mezoth
Aug 7, 2006

Jeff73 posted:

I said I was open to suggestions, not heart attacks. :psyduck:

Disgruntled employees are worse - I worked for a company that laid a bunch of people off during the bubble-pop a few years back, and a line tech walked into a room and cut around 60 voice DS3 connections. It was hell to get fixed from where I was (4 states away) because none of the remaining line-techs cared and the manager was tech incompetent. This helped the company go into bankruptcy even faster!

Honestly, if you do not have a seriously reinforced door on that closet then get a cage installed. Its not as expensive as you think, and keeping junkies/thieves/etc several feet back normally will save your equipment. I cant help you about the 4003's with sup1s however, that is just pushing your luck at this stage. :)

Adbot
ADBOT LOVES YOU

mezoth
Aug 7, 2006

inignot posted:

TACs usefulness is inversely proportional to your level of experience. When you're first starting out & need to know how to get OSPF up on your point to point T1, TAC is seemingly god-like. When you have 10 years of experience and you ask them why your redistribution route map isn't applying tags to all the networks in your prefix list, and all you get from TAC is a blank stare, then you tend to think they suck.

Having worked for them, and then worked for a major internet provider and dealing with them (and the AS group at Cisco) this statement is 100% accurate. Seriously, for advanced routing issues, or getting them to admit to a bug (and internally getting the DEs to admit to a bug) was just a real pain in the rear end without overwhelming evidence. I am tracking multiple bugs with DSCP marking on the 7600 sup720 platform right now, and Cisco will not even try to help anymore. Grr.

For basic/mid-level support, however? They seriously rock. And if they do not, escalate internally and get somebody that does - it shows up in the metrics pretty fast when somebody has a ton of requeues for bad service and action is taken.

mezoth
Aug 7, 2006

GPF posted:

I'm going to come over and rub myself on your boxes. Just fair warning.

Just for my own curiosity, how different is the interface compared to standard IOS?

IOS-XR is usable by anybody that knows IOS, but you are going to get lost pretty easily for a while when actually doing config changes. They basically ripped a bunch of things from JunOS that were good and added them into IOS - modularity, commits on configs, etc. The other major difference that trips most routing types up is the change of route-maps to RPLs - its a very different language then route-maps.

I am just now getting the hang of it, but I have not had a box to seriously play with it on - all of our boxes in the field that use it are production. Thanks, but I prefer not breaking service for customers while playing on a box!

A side note : did you know the CRS 8x10g cards are oversubscribed? Only 40g backplane per linecard!

mezoth
Aug 7, 2006
I have been told by Cisco that IOS-XR is eventually coming for every platform - they are working on the 7600/6500 version right now as a primary task for my company. Commits and commit rollbacks are a sweet sweet thing however!

As for the distributed switching thing - if you have a CRS, you are very very likely to be concerned about more then just one blade of linecards - or even more then one chassis if you really scaled up! And afaik, there is no upgrade available at this time for the fabic on the CRS - if there was, we would probably be using it in several locations.

On the 7600, traffic to the first 4 ports has to hit the backplane to get to the last 4 ports on the card - so it is much less misleading then it appears. I also do not know the 8 port linecards chipset limitations - the 4 port card on the 7600/6500 chassis is not full line rate either, even assuming 100% unicast traffic (19.2gb per 2 ports from the chipset)

mezoth
Aug 7, 2006

jwh posted:

Are you talking about the 6708-10G linecard?

I believe the architecture is like this on the 6708, but I know it is that way on the 6704 - it has to do with the fact that they did not build internal cross connects between the DFC controllers. Since I cannot find the presentation I read on the 6708 to confirm what my memory is telling me, I will accept that I might be wrong on this point. I am trying to remember, but I believe it was one of the key reasons my organization did not go with the 6708 card even for edge routers where we thought we would never hit 10g - backplane congestion is becoming an issue in our more loaded chassis as it is.

mezoth
Aug 7, 2006
Question on uRPF loose mode pertaining to the Cisco platform.

If I have the command "ip verify unicast source reachable-via any allow-default" on an interface, and I am accepting a default route from that interface, it would seem like (from a quick reading of the documentation) I should never get a verify-drop on that interface. However, my understanding is that if any specific match resolves to a null0 route on the Cisco routers, it will drop that route properly - and this matches what I see in my production routers, as I see verify-drops and suppressed drops quite often on those interfaces.

Does anybody know of documentation to back this up, or am I wrong on my presumption on why I am seeing those verify-drops?

mezoth
Aug 7, 2006

jbiel posted:

The reason I ran into it was because another admin turned up a bunch of servers without configuring switchport security. Attempt to use sticky pulled no MAC, and MAC-Address table had no MACs for those ports.

So at that point, I have to bug the sysadmin goofs to get me MACs off the servers, which can be another pain in the rear end in itself.

So being able to see the MAC of the device plugged into the port at all times, regardless of traffic on the port would be nice.

I guess it is just a personal gripe more than anything, it just makes it useful in my situation where we are required to use switchport security.

Ping the subnet that the servers are on?

That should populate the table for you, and you do not have to bug anybody else!

Its expected behavior that was designed around legacy deployments. To change the behavior today would cause more harm then good, as it causes very little harm today. If you have to keep a manifest of what is on the port, use switchport security and sticky to track things, or use eou (Extensible Authentication Protocol over UDP) clientless with a centralized server (typically a radius set to always send back an allow but keep a table of the macs in the eou messages).

mezoth
Aug 7, 2006

Tab8715 posted:

Well, I went school but I never graduated but I could see how getting at least a minor CS would be helpful. If I went for a major in CS I don't see why anyone would bother going into Cisco stuff.


It depends on what the company needs. I would give my left nut right now for a really competent programmer that understands OSPF/BGP and routers really well. For larger networks, there is a strong trend towards automation in the management of the network, and as there are probably no off the shelf solutions that will fit "your" network you will have to build something in house.

Of course, one of the best CCIE's that I worked with had a major in aerospace engineering. It really is not what your degree is in (or if you have a degree, as I do not) but how you apply yourself to the job. I do strongly believe that some CS background is generally useful for troubleshooting computers and anything computer related, just so you get a feel for the underlying methodology of how programs work.

As for physically intensive, it is not really. I run the lab for my group, and the heaviest things that I have to move around on a routine basis are blade-servers. I racked an empty 7609 by myself without too much trouble, but it was going on the bottom of the rack so I cheated. If it is heavier, you normally just grab a coworker and make them help you (or in the case of a CRS-16, the union guys so if they drop it you can blame them).

Now, I am curious what program people use to generate network traffic (preferably something with replay capability and 1g throughput) in their test labs? I have an IXIA chassis, but not the license for the latest software that would give me the ability to generate specific payloads - finance is too cheap at the moment to let me upgrade.

mezoth
Aug 7, 2006
So the IXIA chassis and software that I am currently using seems to be able to generate traffic and routes along arbitrary guidelines, but (and I may just be stupid) it does not seem to be able to replay TCP sessions or even generate specific payloads for packets. They are cannons too, we have some 10g cards for them and they actually do line rate 10g. My understanding is that there is a software upgrade that permits the IXIA chassis to do part of what I want (specific data segments in packets of arbitrary type) but not the TCP replay capability.

My ideal solution would be a medium performance software package (up to 100k pps) with tcp replay capability on some sort of generic hardware platform running linux. It would also be able to rewrite the IP header at replay-time (for things like many generic source-addresses) and preferably intelligently rewrite the layer4 header as well (new tcp seq# that increment along the replay).

To bring this back on topic, the reason I am looking for this is that I am doing some performance testing on 7600 and CRS in relation to the routing engine protection mechanisms. Sadly, both of these products have relatively immature routing engine protection mechanisms and I am finding bugs in them - but some bugs are hard to find without realistic looking traffic to simulate an "attack".

As a side note, the juniper platform's RE protection methods are far more robust (but they also have their caveats!).

mezoth
Aug 7, 2006
jwh, COPP is the 7600 mechanism, and I forget the acronym they use for the CRS - functionally the same in the end, just a slightly different mechanism (and only available in 3.6.0 and later).

Power, it is one of the big ISPs, but I will actually not say which one - being publicly associated with a specific ISP just leads to trouble, either people wanting things/info that you cannot give or hating you for some imagined slight that you had no control over. :\

I can talk about my experiences with COPP in detail, if desired! It definitely has some issues still, but the newer code (SRC+) seems to have ironed some of them out. That is the code I am trying to complete my testing on currently for COPP and why I am looking to generate traffic. The CRS I am just now implementing the protections on as the mechanisms are brand new, and it needs the same gamut of tests.

mezoth
Aug 7, 2006

Tony Montana posted:

Ok, thanks.

The jist of that RFC seemed (to me) to be that an attacker could spoof a private network address as the source of their packets, which would then hit the routing tables inside the target's router and be routed to the host they want to compromise.

But would their ISP route packets with a source in the private network range? Maybe the attacker owns their own links, in Russia or something and doesn't have to go through an ISP. But wouldn't my ISP have private network filtering in place as a precaution anyway? Are you suggesting that even though my ISP probably does, just assuming that isn't the smartest security policy?

I may not understand this at all, be gentle please lol

You *could* trust the upstream is doing this filter, and is watching out for you. Of course, you can trust that every link on here is work safe, and that Bob from accounting packed your parachute correctly.

Honestly, the higher up you go in the aggregation layer the more expensive (in CPU/ASIC resources) it becomes to do filters like this. Now, high end routers say they are line rate on filtering ACLs, but there are still limits on many of them - so normally you want to filter at the edges, especially the smaller link size edges. There are valid reasons that your upstream may not drop the packets, and incompetence is only one of them.

mezoth
Aug 7, 2006

jwh posted:




This topology should work fine, assuming you have nothing special in your configuration for the ports - the layer2 ports should be on a vlan, the layer3 port is just a routed interface (as your diagram indicates). Even assuming the router is trying to populate the cam-table with its own mac-address from the l3 port (something you could test by changing the mac-address of the l3 port) and failing, the worst case that should happen is that all traffic to your gateway floods instead of gets switched - it should not fail.

Being that the box you are evaluating is a DPI acceleration box, I would think it has no issues, but you could always throw another l2 switch in its place to validate the issues are not it and belong to the topology.

I think everything I said holds true if you are not using a sup2 and not a sup720, but it has been a few years since I did extensive testing there. Thinking about it, I am pretty sure you are going to need to change the mac-address of the routed port however to prevent the flooding from happening.

mezoth
Aug 7, 2006

jwh posted:

We actually crossed over Gig8/45 to Gig8/47 and we see the same issue. It's really confounding.

I'm going to try and look into it further this week, but in the interim we got our Steelhead 5050 up and running by putting an 1841 in between it and Gig8/45. Adds some complexity (and there are throughput concerns now), but at least we can get the thing working.

If you are running a sup720, I would suggest trying the mac-address change for the routed interface. If anything, the router is probably getting tripped up on seeing its own mac-address in the CAM table. If I might ask, what code version are you running?

mezoth
Aug 7, 2006

jwh posted:

Yeah, Sup720. I'll try the mac-address change.

Code is 12.2(18)SXE1

Also, i assume the port is not err-disabling? You might have to do "no keepalives" and "no cdp" on the port as well.

mezoth
Aug 7, 2006

ior posted:

Look what I just got in my LAB, 1 x OC768 (40G) and 2 x 8 10GbE. Going to be used at 'The Gathering' terminating the 30Gb/s internet connection :)



It shouldn't be a problem, but make sure you are aware that the 8x10ge is oversubscribed to the fabric. That has bit my company in the rear end a couple of times already!

mezoth
Aug 7, 2006

Mierdaan posted:

Yeah, that's how I was imagining it working from the really brief talk we had today - ethernet handoff to our core L3 switch and the branch office gets its own vlan/vlans. I'll make sure to ask about point-to-multipoint in case we need to do this again in the future.

Thanks for the input guys - goons rule. I am not a network guy, so while I could make something work, not pissing off the next guy or hacking together something unsustainable is important to me.

Just as a side note to this: last time I talked to the Comcast Metro-E guys, it was a fiberline service with a real SLA, and the l2 handoffs were running VPLS in the Comcast network - so you could do multipoint layer2 assuming all sites were in the fiber footprint. They also worked really hard to distance themselves from any of the cable modem services and offer competitive Metro-E services with actual SLAs.

mezoth
Aug 7, 2006

BelDin posted:

The traffic with jumbo MTU going to a switch without it would be the problem as far as I can tell. My understanding is that fragmentation doesn't happen at L2, the frame is just dropped as too large.

The distribution switches are 1500, so if I enable jumbo on them, all 1g and 10g links would have that MTU. Then I would have to change on the L2 switches downstream, etc.

Remember, if you do not set your MTU on the NIC for the server it will not send out packets above 1500 anyway. If you did set it higher then 1500 on the server, then the packets would drop at the first switch that did not have jumbo frames enabled - be it the nexus or the next switch in line. The few NICs that "auto-discover" MTU all do path-mtu-discovery to at least the first L3 hop - so it will max out at 1500 if your router is still only 1500 MTU.

mezoth
Aug 7, 2006

BelDin posted:

Ok, I'm officially stupid. I've been focused on deploying this for so long that it didn't occur to me to just control the frame size at the host level. Will CDP throw errors that I have to supress? Also, I thought that mtu mismatches were detected and ok on layer 3 links, not layer 2.

MTU mismatchs will always cause a frame drop - every "layer 3" link is *also* a layer2 link, never forget that. At a "layer 3" router boundary, the router can fragment the packet if the outbound interface that it is sending the packet on has a lower configured MTU then the packet size it is trying to send (see: ATM). Technically, this is true for a host as well - we just never consider this as it is part of the normal behavior for an endpoint.

Also, unless the host sends out (properly formatted) CDP messages, you will no get any CDP log messages. All mismatch log messages from CDP are a result of receiving a CDP packet on the interface reporting the error that the router about the configuration of the other side of the link.

I do not know about the nexus platform, but on IOS platforms if you enabled "jumbo" frames it actually did not raise the outbound MTU for packets generated / routed by that device, it only allowed the router/switch to accept and switch packets above the configured MTU up to the jumbo maximum for that platform. This methodology was very confusing to people, however.

mezoth
Aug 7, 2006

ragzilla posted:

Still only 32 bits of ASN space. And multi homing should be easier under v6 (just have an address from each provider on your machine) if you're an eyeball.

Prefix delegation is the IPv6 solution for single homed customers that want to run more then one network segment locally, but as far as I know people are talking about doing BGP if you are trying to run to two ISPs in an active/active scenario.

The real issue comes back to DNS resolution - the second you setup a server that you want to be resolvable in global DNS, you really need to know that you can keep that IP address that you have in DNS or that it can be updated very quickly to avoid downtime. DDNS has the downside of low TTL values, so that puts more stress on your auth DNS servers - especially if you are talking about doing this with thousands or tens of thousands of endpoints.

mezoth
Aug 7, 2006

jwh posted:

This made me laugh out loud more than you could imagine.

Some interesting news on this one- my TAC engineer called me back to inform me that, yes, the ASA interface is being overwhelmed by the traffic, and that I have two options, more or less:

a). Stop sending so much traffic

b). Consider port-channels to try and provide more ingress buffers to the platform.

So there you have it! :toot:

This is basically the response TAC gave me for my 5540 overrun problems (constant 3-5% overrun with a max of 250meg traffic on a 1gig interface). In short, the ASA interfaces are pieces of junk. Thanks for letting me know this still happens on the newer line of ASAs, however!

Adbot
ADBOT LOVES YOU

mezoth
Aug 7, 2006

CaptainGimpy posted:


I'm preparing to do an overhaul in a facility with roughly 1200 nodes. I'm looking for recommendations on access layer switches that do gig POE and an option for 10 gig uplinks. Core/Distribution will be Cisco, though willing to look at others. I've had a nasty time with H3C/HP, mainly around their distribution and support model.


There are lots of options in gig POE switches, but I will comment that you need to be aware of what POE devices you are plugging into the switches - if you have older Cisco phones they become a real bitch to manage as they do not support automatic POE on non-cisco switches all of the time. We have that problem where I am currently at, and they are discussing going with HP switches for the edge.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply