Search Amazon.com:
Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us $3,400 per month for bandwidth bills alone, and since we don't believe in shoving popup ads to our registered users, we try to make the money back through forum registrations.
  • Post
  • Reply
Pussy Noise
Aug 1, 2003



mezoth posted:

commits on configs

Yes, please!!!

Adbot
ADBOT LOVES YOU

Pussy Noise
Aug 1, 2003



CrazyLittle posted:

Yeah, why on earth would (or do) carriers put business customers together within the same broadcast domain. Isn't that just a recipe for disaster?

No kidding. I think you guys ought to consider switching providers unless nothing else is available, in which case a long conversation with your account manager at the current provider might be in order...

Pussy Noise
Aug 1, 2003



One of our PE routers is a Cisco 7206VXR (NPE300) running 12.2(25)S10, and we have a couple of customers whose VPN solution requires multiple VRFs per site. An example CE is a Cisco 1841 running 12.4(1c), and their connection is a G.SHDSL via the router's ATM interface, via a third party ATM pipe, to our PE. Now, because it's ATM, we can't just run a VLAN for each VRF, so we tunnel the VRFs.

Anticipating a swap to an Ethernet DSLAM, we prepared the CE for RFC2684 by adding the atm route-bridge ip statement under the CE's ATM point-to-point subinterface, and same on the PE's subinterface pending the swap to Ethernet. Suddenly, the tunnel interface on the PE goes up/down and doesn't come up until we reload the PE no matter how much we wank it. Even then the tunnel interface only stays up for a few hours at a time, and this is a really loving annoying situation.

The corresponding tunnel interface on the PE is up/up the whole time, and the loopback interface on the PE and the CE acting as tunnel sources and destinations are up/up. The loopbacks are routed in the main VRF (customer-office). The really weird thing is that you can't ping them from each other although the routes are visible and the point-to-point connection between the PE and CE ATM subinterfaces pings just fine. All other networks routed in the customer-office VRF ping too, just not the loopback /32 prefixes.

The kludge is implemented like so (using a DMZ VRF as an example):

-- PE --
code:
interface ATM2/0.801 point-to-point
 description CUSTOMER
 ip vrf forwarding customer-office
 ip address 172.16.P.Q 255.255.255.252
 atm route-bridged ip
 no atm enable-ilmi-trap
 pvc 19/73
  vbr-nrt 2048 2048 25
  encapsulation aal5snap
 !
end

interface Tunnel11
 description customer-dmz
 ip vrf forwarding customer-dmz
 ip address 172.16.X.Y 255.255.255.252
 no ip directed-broadcast
 ip mtu 1500
 no clns route-cache
 tunnel source Loopback11
 tunnel destination 192.168.A.B
 tunnel vrf customer-office
end

interface Loopback11
 description customer-dmz loopback
 ip vrf forwarding customer-office
 ip address 192.168.A.C 255.255.255.255
 no ip directed-broadcast
 no clns route-cache
end
-- CE --
code:
interface ATM0/0/0
 description yaddayadda
 no ip address
 no atm ilmi-keepalive
end

interface ATM0/0/0.1 point-to-point
 ip address 172.16.P.R 255.255.255.252
 atm route-bridged ip
 pvc 0/100
  tx-ring-limit 2
  encapsulation aal5snap
 !
end

interface Tunnel5
 description DMZ WAN
 ip vrf forwarding dmz
 ip address 172.16.X.Z 255.255.255.252
 ip mtu 1500
 tunnel source Loopback5
 tunnel destination 192.168.A.C
end

interface Loopback5
 ip address 192.168.A.B 255.255.255.255
end
I worked around this by just terminating the tunnels on one of our bad-rear end Juniper PEs, so this question is mainly academic, but it drives me nuts that I have no idea what might have caused this and if the same gently caress-up is to be expected in the future. There's nothing in the PE log besides the UPDOWN notifications, and nothing at all in the CE log. I'm considering submitting a TAC request, but is there anything I'm overlooking here?

Pussy Noise
Aug 1, 2003



jwh posted:

Yikes, I don't have any ideas- but wouldn't it be easier to provision two PVCs to the same CE instead of working through the GRE vrf forwarding and tunnel vrf stuff? That seems really, really complicated.

The local operator doesn't support that for *DSL connections. Otherwise we would, yeah.

Pussy Noise
Aug 1, 2003



Oh boy have I got a stumper.



PE is Cisco 7206VXR, IOS (C7200-P-M), Version 12.2(25)S
CE is Cisco 2821, IOS (C2800NM-SPSERVICESK9-M), Version 12.4(5a)

The three VLANs are used for three separate MPLS VPNs, respective subinterfaces are directly connected via /30 point-to-point networks. Right now we have a static route setup going, but the customer needs a backup connection, so I'm setting up BGP peering between PE and CE. Evertything's fine except the BGP session across VLAN 212, because a TCP connection can't be established between the point-to-point addresses.

The point-to-point networks for VLANs 210 and 211 are just fine, ICMP goes both ways and BGP sessions are established. There are no access lists, the switch has all three VLANs allowed in its trunk ports, and I can see both ends of the point-to-point networks in both ARP tables for VLAN 212:
code:
PE#sh ip arp vrf customer-vpn-3 Gi0/1.212
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  172.16.36.237           -   aaaa.bbbb.cccc  ARPA   GigabitEthernet0/1.212
Internet  172.16.36.238           0   xxxx.yyyy.zzzz  ARPA   GigabitEthernet0/1.212

CE#sh ip arp vrf vpn-3 Gi0/0.212
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  172.16.36.237           0   aaaa.bbbb.cccc  ARPA   GigabitEthernet0/0.212
Internet  172.16.36.238           -   xxxx.yyyy.zzzz  ARPA   GigabitEthernet0/0.212
However:
code:
CE#ping vrf vpn-3 172.16.36.237 source Gi0/0.212

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.36.237, timeout is 2 seconds:
Packet sent with a source address of 172.16.36.238
.....
Success rate is 0 percent (0/5)
I've verified with the customer that all traffic in that VPN flows fine. Just for shits and giggles, let's ping any address that's elsewhere in the VPN, e.g.:
code:
CE#ping vrf vpn-3 172.16.36.94 source Gi0/0.212

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.36.94, timeout is 2 seconds:
Packet sent with a source address of 172.16.36.238
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/10/12 ms
What the gently caress. I've shut/no shut the subinterfaces n+1 times, there is nothing wrong with the configuration, and I've even reloaded CE. There's nothing in the log of either PE or CE that would indicate anything weird at all. I'm guessing I might have to reload PE (uptime is 2 years, 7 weeks, 9 hours, 54 minutes), but it's not like that's something we want to do willy-nilly.

Here's the interface configuration, as straightforward as it gets:

PE
code:
interface GigabitEthernet0/1.210
 description XXXX
 encapsulation dot1Q 210
 ip vrf forwarding customer-vpn-1
 ip address 172.16.36.229 255.255.255.252
!
interface GigabitEthernet0/1.211
 description XXXX
 encapsulation dot1Q 211
 ip vrf forwarding customer-vpn-2
 ip address 172.16.36.233 255.255.255.252
!
interface GigabitEthernet0/1.212
 description XXXX
 encapsulation dot1Q 212
 ip vrf forwarding customer-vpn-3
 ip address 172.16.36.237 255.255.255.252
!
CE
code:
interface GigabitEthernet0/0.210
 description XXXX
 encapsulation dot1Q 210
 ip address 172.16.36.230 255.255.255.252
 no snmp trap link-status
!
interface GigabitEthernet0/0.211
 description XXXX
 encapsulation dot1Q 211
 ip vrf forwarding vpn-2
 ip address 172.16.36.234 255.255.255.252
 no snmp trap link-status
!
interface GigabitEthernet0/0.212
 description XXXX
 encapsulation dot1Q 212
 ip vrf forwarding vpn-3
 ip address 172.16.36.238 255.255.255.252
 no snmp trap link-status
!
he;lp

Pussy Noise
Aug 1, 2003



inignot posted:

Irrespective of the failed ping, does the address arp successfully?

Yeah, there's constant heavy traffic across the link.

jwh posted:

If you debug icmp on the PE router, do you see echo requests arriving over vlan 212 when you conduct your ping? Also what's CEF say about the 172.16.36.238 adjacency?

Echo replies are logged on PE when pinging it from CE; nothing on CE when pinging it from PE.

CEF says
code:
PE#sh ip cef vrf customer-vpn-3 g0/1.212
---8<---
172.16.36.236/30     attached             GigabitEthernet0/1.212
---8<---
PE#sh ip cef vrf customer-vpn-3 172.16.36.238
172.16.36.238/32
  attached to GigabitEthernet0/1.212

CE#sh ip cef vrf vpn-3 g0/0.212
Prefix              Next Hop             Interface
0.0.0.0/0           172.16.36.237        GigabitEthernet0/0.212
172.16.36.236/30    attached             GigabitEthernet0/0.212
172.16.36.237/32    172.16.36.237        GigabitEthernet0/0.212
CE#sh ip cef vrf vpn-3 172.16.36.237
172.16.36.237/32, version 94, epoch 0, connected, cached adjacency 172.16.36.237
0 packets, 0 bytes
  via 172.16.36.237, GigabitEthernet0/0.212, 1 dependency
    next hop 172.16.36.237, GigabitEthernet0/0.212
    valid cached adjacency

Pussy Noise
Aug 1, 2003



jwh posted:

Yucky. I'm sorry man, it sounds like you're running into a bug of some sort.

Have you tried putting Gig0/1.212 on PE into another VRF, and then putting it back into vpn-3?

Yeah, I kind of think so too Thanks tho'! I haven't tried that yet, but I will on Sunday. The circuit is in production so I can't gently caress with it too disruptively during the week.

Pussy Noise
Aug 1, 2003



I don't have any general answer, but I just checked on my plaything 1841, and it can do PGM Router Assist. It's running IOS version 12.4(5), advipservices feature set. I bet you could get a 1841 for significantly cheaper than HK$74,100.

Have you looked at the Cisco feature navigator at http://tools.cisco.com/ITDIT/CFN/jsp/index.jsp?

Pussy Noise
Aug 1, 2003



Is there a way to debug UDP packets so that I can see which VLAN/subinterface they are received on? The router is a 2611XM, IOS 12.3(9a).

Pussy Noise
Aug 1, 2003



routenull0 posted:

Build an ACL on the interfaces if you are looking for specific traffic. A bit more info on what you are looking for and I can be of more assistance.

The router is a CE with several VRF-lites on separate vlans towards the customer LAN. One LAN segment needs a DHCP pool, so I have it on the router associated with the appropriate VRF. However, the customer's DHCP requests time out, and I don't know why, so I want to blame their LAN Procurve by showing that I'm not seeing any UDP broadcasts on that specific vlan.

Pussy Noise
Aug 1, 2003



Anyone else excited about the Richards Zeta acquisition and EnergyWise? Any idea when 12.2(50)SE will be released?

Pussy Noise
Aug 1, 2003



I just started at a new company in charge of IP networks. The company is a startup and they haven't had a professional network admin before, so bear with me...

Going through all our devices, I noticed some extremely weird poo poo on one CE router. Hardware is 871W, IOS is C870-ADVSECURITYK9-M, Version 12.4(24)T. The log is filled with stuff like this:

code:
Nov  5 04:48:17.791: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
Nov  5 06:30:59.388: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
Nov  5 07:15:01.860: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection
We aren't running any automatic monitoring or CVS system like Rancid that opens SSH sessions (yet), so there should be no reason for log entries like these. Alas, I noticed some nonexistent users from IPs in countries like Poland and Ukraine, e.g.:

code:
  User: lin, line vty 3, service VTY
        Active time 00:00:01
  Timeouts:    Limit     Remaining Timer Type
               00:10:00  00:09:58  Idle Exec
  VTY: Line 3, remote x.x.x.x
  Line: Baud rate (TX/RX) is 9600/9600
  Status: Ready, Active, No Exit Banner
  Modem State: Ready
Other unauthorized usernames included "leandro" and "library". None of these usernames are configured on the router, and we have no remote authentication (yet).

I secured the router with a nazi access list and fired off some abuse reports, so things are under control now. However, going through recent security advisories, I found nothing pertaining. WTF?

Pussy Noise fucked around with this message at Nov 5, 2009 around 09:13

Pussy Noise
Aug 1, 2003



Powercrazy posted:

I assume some type of SSHv1 exploit?

Yeah, it has to be. I guess nobody here figured that it might be a bad idea to run a router with a public interface without access control on the vty

Pussy Noise
Aug 1, 2003



I'm running a WS-C6509-E running (s72033_rp-IPSERVICESK9_WAN-VM), Version 12.2(33)SXH4, and the following modules installed:
code:
  1   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     XXX
  2   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     XXX
  3    6  Firewall Module                        WS-SVC-FWM-1       XXX
  5    5  Supervisor Engine 720 10GE (Active)    VS-S720-10G        XXX
I just got back from the data center after a catastrophic failure trying to add another WS-X6748-GE-TX.

After inserting the new module, the other two switch modules went down (i.e. powered off) and the switch barfed a bunch of crap in the console that looked like a hardware failure.

I removed the new module and powered the two old ones back on, and all the links came up. So why is my network still down? Turns out the internal etherchannel between the FWSM and the switch was hosed.

Reloading the FWSM gave me the following unusual log entries right after passing diagnostics:
code:
Apr 14 19:29:59.612: %EC-SP-5-CANNOT_BUNDLE2: Gi3/1 is not compatible with Po307 and will be suspended (vlan mask is different)
Apr 14 19:29:59.616: %EC-SP-5-CANNOT_BUNDLE2: Gi3/2 is not compatible with Po307 and will be suspended (vlan mask is different)
Apr 14 19:29:59.916: %EC-SP-5-CANNOT_BUNDLE2: Gi3/3 is not compatible with Po307 and will be suspended (vlan mask is different)
Apr 14 19:29:59.920: %EC-SP-5-CANNOT_BUNDLE2: Gi3/4 is not compatible with Po307 and will be suspended (vlan mask is different)
Apr 14 19:29:59.920: %EC-SP-5-CANNOT_BUNDLE2: Gi3/5 is not compatible with Po307 and will be suspended (vlan mask is different)
Apr 14 19:30:01.211: %EC-SP-5-CANNOT_BUNDLE2: Gi3/6 is not compatible with Po307 and will be suspended (vlan mask is different)
Reloading the whole box and then re-reloading the FWSM fixed the problem, but I have no idea what happened. How is this possible on the internal EC, and why did the problem persist after a complete reload? Did something crazy happen to vlan.dat when I inserted the card? Have I missed something obvious in the release notes? Is there anything I can do to prevent this in the future?

I swear to god I'll moonwalk all the way to the data center when our new Juniper SRX is delivered.

Pussy Noise
Aug 1, 2003



Thanks for the help. I can't find a crash file. The whole switch didn't crash, it was just the two line cards I think, and I was able to power them back up from the terminal. Anyways, do you think the new card is faulty, and do you know of any weird Cisco secret crash dump hideouts or histories where I could get some more information on what exactly went wrong when I installed the card?

Maintenance is obviously done during a maintenance window, but it's still desirable to minimize downtime. And yeah yeah all vendors are the same, but I haven't had nearly as many frustrations with Juniper equipment as I've had with Cisco despite working with both about equally. Oh well.

Pussy Noise
Aug 1, 2003



I'm having another strange FWSM issue. We got a Juniper SRX and are migrating customers to it one by one.

Our core services such as DHCP, DNS etc. are terminated on the FWSM, and so far we have four customer networks terminated on the SRX. I use a temporary point-to-point VLAN between the FWSM and the SRX to pass traffic between the customers on the SRX and services on the FWSM.

code:
(DHCP Server)===vlan 3===(FWSM)===vlan 50===(SRX)===vlan 19===(DHCP client)
Right now I'm having an issue with DHCP relay from a customer network whose default gateway is on the SRX. Here's what happens
  1. the host sends a DHCPDISCOVER broadcast
  2. the broadcast hits the SRX (vlan 19), which relays the DHCPDISCOVER to the DHCP server using its loopback address as the source address
  3. the DHCPDISCOVER enters the FWSM through the point-to-point interface (vlan 50)
  4. the DHCPDISCOVER is forwarded out the core services interface (vlan 3) towards the DHCP server
  5. the server sees the DHCPDISCOVER and responds with a DHCPOFFER to the giaddr
  6. the DHCPOFFER enters the FWSM through the core services interface (vlan 3)
  7. ... *poof*, it's gone, no DHCPOFFER leaves the FWSM on any interface, and no conn is created for the packet
Is it possible that the FWSM blocks the DHCPOFFER because the DHCPDISCOVER has the SRX loopback address as its source, whereas the DHCPOFFER has the giaddr as its destination? If so, why does DHCP relay work just fine for the three other networks, which are identically configured? The DHCPOFFER is forwarded out the point-to-point interface, hits the SRX, is relayed to the client, and then a similar sequence with DHCPREQUEST and DHCPACK can be seen.

I don't think it's an xlate issue, or at least clearing the relevant xlates doesn't change the situation. So why does my FWSM eat my DHCP packets? Why is there nothing at all in logs about any of this?

Pussy Noise fucked around with this message at Jun 15, 2010 around 08:57

Pussy Noise
Aug 1, 2003



Peanutmonger posted:

We had problems with FWSMs dropping packets with option-82 information that was fixed in 4.x with the "dhcprelay information trust-all" command. Wouldn't be surprised if they have "features" that cause other DHCP packets to get dropped as well.

Thanks for the response. I hate all these features on the FWSM that you can't loving disable or enable or do anything about. In any case, I finally managed to find a consistent log message that correlates with the dropped DHCPOFFER, and it's:
code:
DHCPRA: dhcp_relay_agent_receiver:can't find binding
ok great but why does it work for the three other VLANs and not for this one where the configuration and packet captures are identical?!

I so do not trust this box anymore and am really happy with the way the SRX deployment is panning out, but it looks like the FWSM is not about to let go of its customers without a fight..

Pussy Noise
Aug 1, 2003



Haha, our SCCM server suddenly stopped being able to connect to any hosts on a certain network segment this morning, and literally nothing helped short of reloading the FWSM. Die.

Pussy Noise
Aug 1, 2003



Not a Cisco-specific question, but I'm hoping a thread full of network pros might be able to help me out here

We are upgrading our datacenter to 10G this summer, and Arista 7100T series looks like a winning proposition for racktop/access layer based on features and price.

On paper they support everything we need, but I have zero experience with Arista and EOS, and was wondering if anyone here has any experiences with their boxes, especially in real-world deployments.

Adbot
ADBOT LOVES YOU

Pussy Noise
Aug 1, 2003



Thanks for the Arista tips. I have a meeting with their local rep on Thursday, let's see what they got

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply