Search Amazon.com:
Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us $3,400 per month for bandwidth bills alone, and since we don't believe in shoving popup ads to our registered users, we try to make the money back through forum registrations.
«370 »
  • Post
  • Reply
Sepist
Dec 25, 2005

FUCK BITCHES, ROUTE PACKETS


Gravy Boat 2k

gooby pls posted:

Any good way to get auto qos on a port channel on a 4500x? I ran auto qos on an unused port, copied the input service policy generated to the port channel interface and the output policy to the member interfaces but the output policy command doesn't seem to have stayed on the member interfaces.

On some switches/linecards you're not able to apply a hardware QoS policy on an interface sharing an ASIC with another interface that has a different QoS policy. Do any other interfaces have a output policy applied?

You can also try shutting down the other ports in the bundle, changing this one and seeing if it sticks.

Adbot
ADBOT LOVES YOU

Methanar
Sep 26, 2013

There will be no mercy


Asking for a friend: how do you handle ddos attacks. Monitoring, seeing who is sending what where, detection, mitigation, etc. I've got some fairly basic edge ACLs set to filter out fragments, NTP/DNS traffic from sources that are not white listed, weird things like GRE and SNPP that I've seen sent my way. 99% of the attacks I've seen haven't necessarily been bandwidth saturating, but a large number of small packets.

None of this actually helps me though when someone is sending 443 at my internet facing LBs and fall over when they try to negotiate 15k DH key exchanges at once or whatever.

Being able to tolerate the internet throwing garbage traffic at you has to be solved problem.

Slickdrac
Oct 5, 2007

Keeper of the Secret


Methanar posted:

Asking for a friend: how do you handle ddos attacks. Monitoring, seeing who is sending what where, detection, mitigation, etc. I've got some fairly basic edge ACLs set to filter out fragments, NTP/DNS traffic from sources that are not white listed, weird things like GRE and SNPP that I've seen sent my way. 99% of the attacks I've seen haven't necessarily been bandwidth saturating, but a large number of small packets.

None of this actually helps me though when someone is sending 443 at my internet facing LBs and fall over when they try to negotiate 15k DH key exchanges at once or whatever.

Being able to tolerate the internet throwing garbage traffic at you has to be solved problem.

Get a good heuristic IPS in front of your network and throw more power at the servers/proxies, basically. Otherwise, just ride it out and DO NOT EVER mention anything publicly about it being in progress unless you absolutely have to. There's not much else to be done without global ISPs removing their collective heads from their asses and doing something to seriously help with security working together.

tortilla_chip
Jun 13, 2007


State exhaustion attacks will always boil down to the weakest link in your service offering. This is typically going to be firewalls/load balancers/ other stateful devices that are in the "normal" traffic path. You can redirect traffic upstream to offload the state problem (these are your typical scrubbing services (Prolexic, Arbor, etc.).

Sepist
Dec 25, 2005

FUCK BITCHES, ROUTE PACKETS


Gravy Boat 2k

Methanar posted:

None of this actually helps me though when someone is sending 443 at my internet facing LBs and fall over when they try to negotiate 15k DH key exchanges at once or whatever.

Being able to tolerate the internet throwing garbage traffic at you has to be solved problem.

Speaking specifically to this item - on some firewalls (Palo alto and ASA for sure) you can setup a protection policy to perform RED or Syn cookies after a threshold is met on a specific ACL line (EG: After 3k SYN PPS to your LB Gateway, start performing RED on the incoming SYNs until the rate lowers). Being that granular requires you to know what a normal day pps rate is, once you figure that out your golden.

It's still going to degrade performance, but it should keep your poo poo up.

Methanar
Sep 26, 2013

There will be no mercy


Is this sort of thing something that could be handled with a local instance of snort and IPtables? Or is it probably too late by then to mitigate by virtue of the traffic has passed through the network card and the kernel has needed to look at it, even if an https listener didn't.

Buying firewalls that would be capable of processing the amount of traffic I get would be ridiculously expensive. The best scenario would be each internet facing machine is capable of doing it's own filtering.

I've got a lot of things that are internet facing, not just haproxy (although these getting hit particularly sucks and are definitely not IP-portable or acceptable to null route). Some of the machines are pretty massive and are handling at least 1gbps of traffic at any given moment. But it's weird that I know they definitely don't like when someone points a laser at them, even if the attack seems small on the order of a 10k-50k packets per sec.

Partycat
Oct 25, 2004

NO NOT LIKE THE STUPID CAT WITH THE PIPE AND HAT AND SUCH

Plaster Town Cop

Depends. If its targeted at your application then it may be worth spending CPU for minor mitigation. If you're getting 600Mbps+ being tossed at your hardware it may crush the machine and it will be pointless to do it there. The idea is to bounce the session or traffic off of your service somewhere enough to let the legitimate stuff through.

Thus why the suggestion to have your upstream provider can it when its frac on their multiple 100Gbps links before it gets to you.

Methanar
Sep 26, 2013

There will be no mercy




At this moment I have about 20gbps inbound altogether. Highest peak I've ever seen is about 32gbps, although this is growing very quickly. That's my baseline for legitimate traffic. Current CPU utilization is about 50% so maybe you're right, pushing everything through snort would cause everything to grind to a halt, or at least potentially start causing CPU wait.

I don't think upstream filtering is really what I want here. Some of my providers provide blackhole community strings, but ultimately I'd still need to write some sort of script to automatically detect an attack, who it's coming from and then advertise the blackhole string. And that's all assuming the source is well defined and something that's reasonable to block, and not some AWS /12 or ISP's /18 which could have legitimate users. Both of which I've seen garbage originate (or spoofed) from.

Sepist
Dec 25, 2005

FUCK BITCHES, ROUTE PACKETS


Gravy Boat 2k

In the interest of cheapness:

You could put a bump in the wire IPS inline on another server and use some kind of fail open device like a garland bypass switch to allow connectivity to continue if the IPS dies

ate shit on live tv
Feb 15, 2004


For me since our normal traffic looks like DDoS (a shitload of short lived http/s transactions from mostly unique IPs), and we do around 70Gb/s over a global anycast network we just endure it. We've been hit a few times but with only like 10Gb/s which caused some slower then normal responses in a specific region but overall didn't affect us. The things that take us down are DNS provider attacks.

Otherwise just normal edge hardening. Control Plane Policing, ACL's that block everything except 80/443 to our specific service VIPs. We looked into Remote Trigger Blackhole so that we could save our other services, but unfortunately very few providers support it, and those that do only allow you to null route an entire /24, which is the exact opposite of what we want :/

And and yea, almost everything from the attacks we've gotten has been spoofed sources.

ragzilla
Sep 9, 2005
don't ask me, i only work here




ate poo poo on live tv posted:

For me since our normal traffic looks like DDoS (a shitload of short lived http/s transactions from mostly unique IPs), and we do around 70Gb/s over a global anycast network we just endure it. We've been hit a few times but with only like 10Gb/s which caused some slower then normal responses in a specific region but overall didn't affect us. The things that take us down are DNS provider attacks.

Otherwise just normal edge hardening. Control Plane Policing, ACL's that block everything except 80/443 to our specific service VIPs. We looked into Remote Trigger Blackhole so that we could save our other services, but unfortunately very few providers support it, and those that do only allow you to null route an entire /24, which is the exact opposite of what we want :/

And and yea, almost everything from the attacks we've gotten has been spoofed sources.

Who're your current transits? Pretty much everyone I transit with has dRTBH down to a /32.

doomisland
Oct 5, 2004



Trend is to do on host defenses and just scale that out. Servers can do line rate filtering at 10+Gbps via DPDK/eBFP/XDP. and there are a bunch of toolsets out there with that stuff. IPS and other crap for detection of less volumetric attacks. A big to do is to be able to get the appropriate metrics and then be able to product actionable events on the data for Ops folks to do. Just staring at a dashboard for DDoS detection doesn't work well.

Moey
Oct 22, 2010



Anyone running a virtualized firewall in production?

I am replacing a bunch of SRX240 next year, and am eyeballing the SRX345, but now wondering about the vSRX.

I have used the vSRX for test stuff, but no production.

Thanks Ants
May 21, 2004

Bless you, ants. Blants.




Fun Shoe

Is there a historical reason for VLANs in IOS being in their own little file rather than just rolled into the switch config?

jwh
Jun 12, 2002



Thanks Ants posted:

Is there a historical reason for VLANs in IOS being in their own little file rather than just rolled into the switch config?

Yes, it's from CatOS having a vlan.dat.

At least, that was always my understanding.

Nowadays I don't know why they still do it.

Sepist
Dec 25, 2005

FUCK BITCHES, ROUTE PACKETS


Gravy Boat 2k

Ios-xe doesn't have it, and monolithic ios is basically dead now so you shouldn't really ever see it modern hardware

falz
Jan 29, 2005

01100110 01100001 01101100 01111010


Thanks Ants posted:

Is there a historical reason for VLANs in IOS being in their own little file rather than just rolled into the switch config?

It's because vtp is on. switch it to transparent and they'll show in config.

nescience
Jan 24, 2011

h'okay


Partycat posted:

Cucm/jabber contact stuff

Hey thanks, I'll play around with this more and try to use it, I was originally looking for a contact list .XML that could be imported client side, as in from the Cisco Jabber application itself, without access to IM&P server, but this sounds like a better way.

ate shit on live tv
Feb 15, 2004


ragzilla posted:

Who're your current transits? Pretty much everyone I transit with has dRTBH down to a /32.

A little company that is looking to be bought out called Internap.

Adbot
ADBOT LOVES YOU

Kazinsal
Dec 13, 2011



Anyone at Live right now and planning on doing the NetApp FlexPod event?

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply
«370 »