Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
single-mode fiber
Dec 30, 2012

Sepist posted:

Are you using FEX's on your 7k's? Apparently it was a 4-minute full outage but the onsite guy failed to tell me that. They have tested this successfully in the past but they have added FEX since that test. I know the 7k locks up during dual-homed FEX sync, not sure if the 5k does. I hope not.

Edit: Turns out they forgot to run a second trunk cable so they lost all access when the one carrying all the traffic was powered off. Finally, not a cisco bug :toot:

Looks like your problem is resolved but our FEXs are not dual homed, we did the topology where dual uplinks go to different line cards on one 7K chassis, and there's 2 FEXs in top of rack.

Adbot
ADBOT LOVES YOU

Ginger Beer Belly
Aug 18, 2010



Grimey Drawer

quote:

I'm going to be spending a fair bit of time figuring out how to handle DDOSes, load balancing, traffic shaping and so on multihomed with 4 providers with an awful lot of bandwidth.

My understanding is the blunt way of weighting certain links higher than others is AS-path pre-pending your advertisements to different peers. You can also break a /24 into a /26 for more granular control

First off ... AS path prepending is not as granular as you might hope. Many ISPs treat customer announcements with a different local-pref than they treat peer-learned announcements. This means that they'll ignore the AS path length decision point because local pref occurs before that and no amount of prepending will override the local-pref difference.

Most major ISPs in North America provide BGP communities for manipulating the local-pref that they assign to routes learned from customers. Check out https://onestep.net/communities/.

Given that, it sounds to me like you think that your job of handling DDoS attacks will include manipulating BGP announcements to make sure that the attack traffic gets distributed over your 4 transit providers in such a way that you aren't congested. That's not a technique that I generally deal with, because if the attack could threaten one of my links, I just go ahead and engage cloud mitigation options.

Feel free to reach out to me directly and I'd be happy to discuss some strategies that I use for my network that gets targeted a lot.

Ginger Beer Belly fucked around with this message at 02:36 on Oct 7, 2016

FatCow
Apr 22, 2002
I MAP THE FUCK OUT OF PEOPLE

Methanar posted:

I'm going to be spending a fair bit of time figuring out how to handle DDOSes, load balancing, traffic shaping and so on multihomed with 4 providers with an awful lot of bandwidth.

Unless 'an awful lot' is close to 1Tbps, just pay someone to do it for you. Even if you absorb it you still have to do something with the attack to filter out the bad traffic from the good traffic. Will you even be able to get to your routers, do you have a robust OOB network available? If you're running an eyeball network you can likely get away with RTBH with your transits to just dump the traffic on their edges. If you're running a content provider, open up the checkbook.

I have 160Gb/s of transit available to me and about 100Gb/s of peering and I still pay networks much, much larger than me to filter DDoS traffic because I can easily be squashed if someone really wanted to.

Also, most providers won't accept anything smaller than a /24, and if they do they won't advertise it out to other peers.

"Internet Routing Architectures" was a good read for me.

tadashi
Feb 20, 2006

When I create site-to-site VPNs from my ASA 5510, the tunnel will only become active once I send packets from my side to the remote site. This seams like secure behavior but is there a way to just allow traffic to start passing once the tunnel exists?

My long term worry is that, if I had an ASA just like mine at a remote site and wanted to create a site-to-site VPN between them, how would I initiate traffic if both end wants to initiate the traffic first? Maybe I'm over complicating things.

adorai
Nov 2, 2002

10/27/04 Never forget
Grimey Drawer
any interesting traffic from either side should bring the tunnel up.

Slickdrac
Oct 5, 2007

Not allowed to have nice things
Two levels of VPN tunnel. The tunnel itself, and the sessions traversing it.

If the tunnel is not up and something hits wanting to go, tunnel will be established, and a session created across it for the traffic. If something comes the other way, new session within the tunnel will be created and tracked.

If something on both sides wants to get to the other, they will each send out a pole to the other to form the tunnel, whichever end pokes the other first is the initiator, but still the same tunnel gets formed regardless and 2 sessions are made within the tunnel.

Kazinsal
Dec 13, 2011



Does anyone here know how to get useful PCM audio out of a voice hpi capture dump from a CUBE gateway? All the documentation I've seen says "send it to TAC", but what I really want is the actual PCM data from the DSPs.

Bigass Moth
Mar 6, 2004

I joined the #RXT REVOLUTION.
:boom:
he knows...
I believe you can listen to the pcm files in audacity.

Prescription Combs
Apr 20, 2005
   6

tadashi posted:

When I create site-to-site VPNs from my ASA 5510, the tunnel will only become active once I send packets from my side to the remote site. This seams like secure behavior but is there a way to just allow traffic to start passing once the tunnel exists?

My long term worry is that, if I had an ASA just like mine at a remote site and wanted to create a site-to-site VPN between them, how would I initiate traffic if both end wants to initiate the traffic first? Maybe I'm over complicating things.

As already mentioned, if they're both configured then it doesn't matter which side initiates. If you are concerned with making sure the tunnel stays up you could get rid of the idle timer with a group-policy or use an sla monitor to send traffic to an arbitrary address across the tunnel. I usually only see 1-2 packets get dropped while the negotiations occur if the tunnel is not currently established and I generate traffic destined for the remote side.

MrMoo
Sep 14, 2000

FYI: just spotted an outstanding iOS and it appears Sierra feature with IKEv2 and Strongswan with MOBIKE & DPD should be resolved in the next revision of Strongswan (5.5.1):

https://wiki.strongswan.org/projects/strongswan/wiki/AppleClients

Sepist
Dec 26, 2005

FUCK BITCHES, ROUTE PACKETS

Gravy Boat 2k
Does anyone know if the latest deployment of FirePOWER Management Center allows you to make changes to the ASA config, or is it still a segregated piece of poo poo?

Edit: Found it, just in case anyone's wondering the new unified image is called FirePOWER Threat Defense and is replacing the standard os image.

Sepist fucked around with this message at 23:19 on Oct 19, 2016

Wicaeed
Feb 8, 2005
What's a good getting started point on a UCS Certification?

I'm currently looking at the Data Center Unified Computing Implementation class on the Cisco website.

A little background:

I'm 1.3 years into my current role (8 years overall experience) and we have a small UCS cluster running a production VMware environment (30-ish UCS blades, 3 chassis, 2 FI).

In the past, the company has had to outsource simple UCS infrastructure changes (Adding VLANs, creating service profiles) until I got here. I'm comfortable doing potentially non-disruptive changes, but larger things like Firmware updates still kind of frighten me. We actually paid one of our vendors to come on site with a UCS guy and do a firmware update about a year ago, however they were supposed to document the process for us and never did.

My boss has tentatively signed off on some Cisco classes for me, and I'm trying to find out where to start.

I don't have a CCNA, but am comfortable with Cisco networking & networks in general.

psydude
Apr 1, 2008

Sepist posted:

Does anyone know if the latest deployment of FirePOWER Management Center allows you to make changes to the ASA config, or is it still a segregated piece of poo poo?

Edit: Found it, just in case anyone's wondering the new unified image is called FirePOWER Threat Defense and is replacing the standard os image.

FTD only really applies to the newest FP 4100/9300s and ASAs. The previous generation still uses two separate code trains, although the functionality is becoming increasingly intertwined (firepower now triggers failover, etc.). Expect to still be dealing with the ASA OS for a while.

Sepist
Dec 26, 2005

FUCK BITCHES, ROUTE PACKETS

Gravy Boat 2k
Fair enough. I was referencing the X series but it's good to know the non-x don't support unified image.

Ahdinko
Oct 27, 2007

WHAT A LOVELY DAY
I was checking out the FTD image for the ASA's, it seems to increase throughput by 10-20% which I guess is a nice way of getting a bit more out of your firewalls before you need to replace them.

Thanks Ants
May 21, 2004

#essereFerrari


What do people think of the Catalyst 2960-L range for really basic access requirements? I want something that I can use the CLI on but also has a noddy UI for some of the people I work with.

psydude
Apr 1, 2008

Ahdinko posted:

I was checking out the FTD image for the ASA's, it seems to increase throughput by 10-20% which I guess is a nice way of getting a bit more out of your firewalls before you need to replace them.

Be aware that the migration process is a bit cumbersome and requires standing up a second FMC.

psydude fucked around with this message at 14:48 on Oct 28, 2016

Slickdrac
Oct 5, 2007

Not allowed to have nice things
Here's an odd one I can't find any Google answer to.

Got an ASA 5525X VPN'd to a remote ASA 5525X. Everything is great. Today they asked for a new destination to be added for the remote end. We both made the entries in our ACLs, and the new destination is not reachable, and troubleshooting is an odd one.

ACL hitcount IS incrementing.
Phase 2 IS establishing
0 packets encap
Log will tell me all about the Phase 2 establishing fine, but NOTHING about the traffic that's causing it to form.

All the entries that were already in continue to work just fine.

Sepist
Dec 26, 2005

FUCK BITCHES, ROUTE PACKETS

Gravy Boat 2k
Sometimes you have to clear phase 2, because policy VPN's are poo poo

Also if you emulate the packet via packet-tracer command do you see it hit the VPN phase? If not check your nat exemption.

Slickdrac
Oct 5, 2007

Not allowed to have nice things
Killed the tunnel entirely, it reformed as expected with same results.

This is what I get from a bad vs good tracer

Phase: 5
Type: VPN
Subtype: encrypt
Result: DROP
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7fff28fdd490, priority=70, domain=encrypt, deny=false
hits=464, user_data=0x0, cs_id=0x7fff34450f10, reverse, flags=0x0, protocol=0
src ip/id=10.0.0.1, mask=255.255.255.192, port=0, tag=0
dst ip/id=10.1.1.1, mask=255.255.255.255, port=0, tag=0 dscp=0x0
input_ifc=any, output_ifc=OUTSIDE


Phase: 5
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7fff34fa37e0, priority=70, domain=encrypt, deny=false
hits=13, user_data=0x5d2d9c, cs_id=0x7fff3444e3f0, reverse, flags=0x0, protocol=0
src ip/id=10.0.0.1, mask=255.255.255.192, port=0, tag=0
dst ip/id=10.2.2.2, mask=255.255.255.255, port=0, tag=0 dscp=0x0
input_ifc=any, output_ifc=OUTSIDE

Which isn't terribly clear why it's dropping the one? There is no NAT of the backside addresses on this VPN.

Slickdrac fucked around with this message at 20:37 on Oct 28, 2016

Sepist
Dec 26, 2005

FUCK BITCHES, ROUTE PACKETS

Gravy Boat 2k
Looks like the remote end acl is not an exact mirror of your side. Usually it's a mismatched subnet mask.

Slickdrac
Oct 5, 2007

Not allowed to have nice things
As much as I love "blame the distant end", shouldn't either the P2 not establish or the traffic still enter the tunnel anyway and just get stopped at the other end?

Sepist
Dec 26, 2005

FUCK BITCHES, ROUTE PACKETS

Gravy Boat 2k
I'm having a little trouble getting my piece of poo poo VIRL instance running right now to prove it out but with a mismatch it should still come up if there's at least one matched acl. I'll update this if I'm able to get it working..

Edit: Got it to work but couldn't replicate the issue so probably not it. Is it possible you have that destination subnet in another active cryptomap?

Sepist fucked around with this message at 22:40 on Oct 28, 2016

psydude
Apr 1, 2008

Check routing to make sure all devices, including the ASA, have a route pointing to the outside interface. If you're using RRI, make sure the reverse routes are redistributed into any dynamic routing processes.

Slickdrac
Oct 5, 2007

Not allowed to have nice things

Sepist posted:

I'm having a little trouble getting my piece of poo poo VIRL instance running right now to prove it out but with a mismatch it should still come up if there's at least one matched acl. I'll update this if I'm able to get it working..

Edit: Got it to work but couldn't replicate the issue so probably not it. Is it possible you have that destination subnet in another active cryptomap?

Checked that, and routes are all pointing the right way, I removed the line from the ACL and tried things out and it popped into log as expected. Opened a ticket with Cisco, so we'll see if they have any thoughts. Or it might just fix itself magically or "magically" once the guys that own the other side wake up tomorrow and check/bop the tunnel from their end.

falz
Jan 29, 2005

01100110 01100001 01101100 01111010
Dear lord Cisco style crypto maps are the worst. All hail VTI interfaces on routers and the sanity that is actual routed interfaces and IGP.

Contingency
Jun 2, 2007

MURDERER

Slickdrac posted:

Killed the tunnel entirely, it reformed as expected with same results.

This is what I get from a bad vs good tracer

Phase: 5
Type: VPN
Subtype: encrypt
Result: DROP
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7fff28fdd490, priority=70, domain=encrypt, deny=false
hits=464, user_data=0x0, cs_id=0x7fff34450f10, reverse, flags=0x0, protocol=0
src ip/id=10.0.0.1, mask=255.255.255.192, port=0, tag=0
dst ip/id=10.1.1.1, mask=255.255.255.255, port=0, tag=0 dscp=0x0
input_ifc=any, output_ifc=OUTSIDE


Phase: 5
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7fff34fa37e0, priority=70, domain=encrypt, deny=false
hits=13, user_data=0x5d2d9c, cs_id=0x7fff3444e3f0, reverse, flags=0x0, protocol=0
src ip/id=10.0.0.1, mask=255.255.255.192, port=0, tag=0
dst ip/id=10.2.2.2, mask=255.255.255.255, port=0, tag=0 dscp=0x0
input_ifc=any, output_ifc=OUTSIDE

Which isn't terribly clear why it's dropping the one? There is no NAT of the backside addresses on this VPN.

ASA VPN tunnels are built on demand. Receiving a VPN encrypt subtype drop message is expected behavior when there isn't a currently active SA. If you do see that subphase, you've sent interesting traffic which will kickstart the tunnel generation process. You should run packet-tracer 5-10 seconds later to see if the result changes from drop to allow.

If it doesn't, you need to run a debug and pay close attention to the SA in question, not any SA, as establishing a SA between your network and remote network A doesn't automatically include remote network B. Phase 2 errors to look for if your side initiates the tunnel:
Invalid ID info: network mismatch.
No proposal chosen: phase 2 settings mismatch (hashing, encryption, etc). This is unlikely to happen on a tunnel with a already working SA.

If there is a network mismatch, people are bad at ASAs and can't provide correct information. Having them initiate tunnel generation while you run a debug will allow you to observe the networks they are proposing in their SA (look for "proxy") and has the advantage of performing hole matching on your side, so you can verify it is being matched to the correct VPN (overlap scenario like Sepist proposed).

If you've verified a SA can be established using the new network (the second packet-tracer run would confirm), things to check on your side:
1) sysopt connection permit-vpn--if it's disabled, you need to have an ACL entry for their incoming traffic. You will see decaps on the SA stats even if the traffic is dropped by your firewall.
2) Is traffic from your side being routed to the ASA? Run something like "telnet 10.2.2.2 445" on your host (anything that causes a successful connection timeout rather than a fast fail) and see if it makes an entry in your connection table. If not, you should run a traceroute on your host. Remember, VPN traffic is encrypted, so if you see Internet hops, your traffic isn't encrypted/on the VPN. If they send you TCP traffic, a good hint that your reply traffic isn't making it back to them is checking the connection status--I believe an incomplete handshake originating from their side would be SaAB.
3) If you are seeing a VPN subtype phase when testing you>them traffic packet tracer, your NAT is probably set up correctly, but it wouldn't hurt to see if there are any NAT statements in place for remote network A that aren't in place for B, and at the appropriate priority.

Jedi425
Dec 6, 2002

THOU ART THEE ART THOU STICK YOUR HAND IN THE TV DO IT DO IT DO IT

Contingency posted:

ASA VPN tunnels are built on demand. Receiving a VPN encrypt subtype drop message is expected behavior when there isn't a currently active SA. If you do see that subphase, you've sent interesting traffic which will kickstart the tunnel generation process. You should run packet-tracer 5-10 seconds later to see if the result changes from drop to allow.

If it doesn't, you need to run a debug and pay close attention to the SA in question, not any SA, as establishing a SA between your network and remote network A doesn't automatically include remote network B. Phase 2 errors to look for if your side initiates the tunnel:
Invalid ID info: network mismatch.
No proposal chosen: phase 2 settings mismatch (hashing, encryption, etc). This is unlikely to happen on a tunnel with a already working SA.

If there is a network mismatch, people are bad at ASAs and can't provide correct information. Having them initiate tunnel generation while you run a debug will allow you to observe the networks they are proposing in their SA (look for "proxy") and has the advantage of performing hole matching on your side, so you can verify it is being matched to the correct VPN (overlap scenario like Sepist proposed).

If you've verified a SA can be established using the new network (the second packet-tracer run would confirm), things to check on your side:
1) sysopt connection permit-vpn--if it's disabled, you need to have an ACL entry for their incoming traffic. You will see decaps on the SA stats even if the traffic is dropped by your firewall.
2) Is traffic from your side being routed to the ASA? Run something like "telnet 10.2.2.2 445" on your host (anything that causes a successful connection timeout rather than a fast fail) and see if it makes an entry in your connection table. If not, you should run a traceroute on your host. Remember, VPN traffic is encrypted, so if you see Internet hops, your traffic isn't encrypted/on the VPN. If they send you TCP traffic, a good hint that your reply traffic isn't making it back to them is checking the connection status--I believe an incomplete handshake originating from their side would be SaAB.
3) If you are seeing a VPN subtype phase when testing you>them traffic packet tracer, your NAT is probably set up correctly, but it wouldn't hurt to see if there are any NAT statements in place for remote network A that aren't in place for B, and at the appropriate priority.

Protip when debugging if you have multiple VPN tunnels:

debug crypto condition reset (this clears the filter)
debug crypto condition peer A.B.C.D (this filters by the peer IP of the VPN you're working on)

Methanar
Sep 26, 2013

by the sex ghost
Does anyone have any thoughts of running ospfd on a server opposed to a hardware switch? I've got a use case where I think it could save me a bunch of public IPs.

FatCow
Apr 22, 2002
I MAP THE FUCK OUT OF PEOPLE
What's the use case? It could be anything from, yeah "that'll work for your anycast DNS server" to "How much does your position pay, and do I want to work for your company?".

Methanar
Sep 26, 2013

by the sex ghost
That bad, huh?

I think I might have been going about the problem wrong anyway after thinking about it for a couple of hours.

My situation was that I had several different WANs with a couple different edge routers and I thought a real internal routing protocol would be necessary to be able to send traffic to different edge routers depending on where the destination would be. Right now everything has a default route pointing to one edge which is a single point of failure and also just not going to work with what we're going for.

Current idea is to set up a few VRRP instances where each edge has at least one vIP that it's active on and then spread out the vIPs to use across all the servers and let iBGP take care of shuffling traffic around if it needs to go out a WAN the edge with the active vIP didn't have.

I still think it's an interesting question when, if ever, you would want to run a routing protocol right on a server.

tortilla_chip
Jun 13, 2007

k-partite
We run BGP on our servers as a load balancing mechanism.

adorai
Nov 2, 2002

10/27/04 Never forget
Grimey Drawer
My core routing is done with vyos running ospfd. Works great for that use case. Not sure about your specific one, but if you think it will solve a problem, test it out in a lab.

FatCow
Apr 22, 2002
I MAP THE FUCK OUT OF PEOPLE
I'd need a diagram to be sure, but it sounds like you just want a fairly standard OSPF/iBGP setup. Make sure every edge router has a loopback, advertise it into BGP along with all your /30s (Internal and external). Peer all the edge routers in a full iBGP mesh.

Methanar
Sep 26, 2013

by the sex ghost
On the surface it is absolutely a textbook ospf/ibgp case. As I think more and more about it my original idea of running ospf right off of the servers was unnecessary and the IP wasting issue wouldn't have been as bad as I thought. So a normal ospf implementation would been just fine too. Only the networks below the top of racks would need to be public, between top of rack and edge could be 10.255.255.0/24 or whatever. Each rack needs 5-6 public IPs so it wouldn't have been 50% of my /30 IPs going to waste either.

With all that in mind I still think it's much easier and requires less downtime to do a handful of vIPs rather than to completely overhaul the entire network to get OSPF in. Although, if I'm ever going to implement OSPF in this particular datacenter now is absolutely the time to do it.

I'm still thinking about it.

MrMoo
Sep 14, 2000

Netgate appear to have released a new tiny system for upto 300mbps firewalls:



I wish they would swap over to USB-C or micro-USB powering instead of the terrible prop bricks.

Slickdrac
Oct 5, 2007

Not allowed to have nice things
So, we fixed the issue. Cisco didn't understand wtf was wrong with it with full debug going, so the destination IP was changed from 91 to 92 and it just magically worked. No other VPN change on either end.

wolrah
May 8, 2006
what?

MrMoo posted:

Netgate appear to have released a new tiny system for upto 300mbps firewalls:



I wish they would swap over to USB-C or micro-USB powering instead of the terrible prop bricks.

I have a preorder in on one of these. They're really appealing for the customers I have who want to have an extension of their business VoIP system at home, and I love the SG-2xxx series, so I'm hoping it works well.

Agreed on the power connection. At least the console port is a micro this time, the SG-2xxx devices use the old mini-USB connector. Fortunately they have started including cables with the devices, they didn't initially.

It's worth noting that these are internally a router-on-a-stick. The SoC contains a three port ethernet switch with the CPU attached to one of the ports. It doesn't seem like that'll be the limiting factor one way or another, but in the event that somehow pf gets a lot more efficient in the future it'll still be hard capped at 500mbit/sec passing between the interfaces.

Methanar
Sep 26, 2013

by the sex ghost
How can I gracefully cease all traffic on one of my wan links?

I'm thinking I can prevent traffic from travelling outbound on the wan link in question by lowering the local preference relative to every other link:

route-map LOCALPREF permit 10
set local-preference 50
router bgp 1
neighbour 1.1.1.1 route-map LOCALPREF out
Then for a full stop of traffic including inbound I can assign the GSHUT attribute:

route-map GSHUT permit 20
set community GSHUT
router bgp 1
neighbour 1.1.1.1 route-map GSHUT out

My concern here is how does GSHUT work, does the peer wait until it has reconverged a different path to my AS before it removes the route from it's table?

Maybe I should assign a bgp community to lower ISP side local preference and apply a few prepends too, they aren't a guarantee that nobody will go on the link but it will at least have fewer people on it.

Methanar fucked around with this message at 22:02 on Nov 4, 2016

Adbot
ADBOT LOVES YOU

CrazyLittle
Sep 11, 2001





Clapping Larry

Methanar posted:

How can I gracefully cease all traffic on one of my wan links?

Unplug it with a curtsy

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply