Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Sepist
Dec 26, 2005

FUCK BITCHES, ROUTE PACKETS

Gravy Boat 2k

gooby pls posted:

Any good way to get auto qos on a port channel on a 4500x? I ran auto qos on an unused port, copied the input service policy generated to the port channel interface and the output policy to the member interfaces but the output policy command doesn't seem to have stayed on the member interfaces.

On some switches/linecards you're not able to apply a hardware QoS policy on an interface sharing an ASIC with another interface that has a different QoS policy. Do any other interfaces have a output policy applied?

You can also try shutting down the other ports in the bundle, changing this one and seeing if it sticks.

Adbot
ADBOT LOVES YOU

Methanar
Sep 26, 2013

by the sex ghost
Asking for a friend: how do you handle ddos attacks. Monitoring, seeing who is sending what where, detection, mitigation, etc. I've got some fairly basic edge ACLs set to filter out fragments, NTP/DNS traffic from sources that are not white listed, weird things like GRE and SNPP that I've seen sent my way. 99% of the attacks I've seen haven't necessarily been bandwidth saturating, but a large number of small packets.

None of this actually helps me though when someone is sending 443 at my internet facing LBs and fall over when they try to negotiate 15k DH key exchanges at once or whatever.

Being able to tolerate the internet throwing garbage traffic at you has to be solved problem.

Slickdrac
Oct 5, 2007

Not allowed to have nice things

Methanar posted:

Asking for a friend: how do you handle ddos attacks. Monitoring, seeing who is sending what where, detection, mitigation, etc. I've got some fairly basic edge ACLs set to filter out fragments, NTP/DNS traffic from sources that are not white listed, weird things like GRE and SNPP that I've seen sent my way. 99% of the attacks I've seen haven't necessarily been bandwidth saturating, but a large number of small packets.

None of this actually helps me though when someone is sending 443 at my internet facing LBs and fall over when they try to negotiate 15k DH key exchanges at once or whatever.

Being able to tolerate the internet throwing garbage traffic at you has to be solved problem.

Get a good heuristic IPS in front of your network and throw more power at the servers/proxies, basically. Otherwise, just ride it out and DO NOT EVER mention anything publicly about it being in progress unless you absolutely have to. There's not much else to be done without global ISPs removing their collective heads from their asses and doing something to seriously help with security working together.

tortilla_chip
Jun 13, 2007

k-partite
State exhaustion attacks will always boil down to the weakest link in your service offering. This is typically going to be firewalls/load balancers/ other stateful devices that are in the "normal" traffic path. You can redirect traffic upstream to offload the state problem (these are your typical scrubbing services (Prolexic, Arbor, etc.).

Sepist
Dec 26, 2005

FUCK BITCHES, ROUTE PACKETS

Gravy Boat 2k

Methanar posted:

None of this actually helps me though when someone is sending 443 at my internet facing LBs and fall over when they try to negotiate 15k DH key exchanges at once or whatever.

Being able to tolerate the internet throwing garbage traffic at you has to be solved problem.

Speaking specifically to this item - on some firewalls (Palo alto and ASA for sure) you can setup a protection policy to perform RED or Syn cookies after a threshold is met on a specific ACL line (EG: After 3k SYN PPS to your LB Gateway, start performing RED on the incoming SYNs until the rate lowers). Being that granular requires you to know what a normal day pps rate is, once you figure that out your golden.

It's still going to degrade performance, but it should keep your poo poo up.

Methanar
Sep 26, 2013

by the sex ghost
Is this sort of thing something that could be handled with a local instance of snort and IPtables? Or is it probably too late by then to mitigate by virtue of the traffic has passed through the network card and the kernel has needed to look at it, even if an https listener didn't.

Buying firewalls that would be capable of processing the amount of traffic I get would be ridiculously expensive. The best scenario would be each internet facing machine is capable of doing it's own filtering.

I've got a lot of things that are internet facing, not just haproxy (although these getting hit particularly sucks and are definitely not IP-portable or acceptable to null route). Some of the machines are pretty massive and are handling at least 1gbps of traffic at any given moment. But it's weird that I know they definitely don't like when someone points a laser at them, even if the attack seems small on the order of a 10k-50k packets per sec.

Partycat
Oct 25, 2004

Depends. If its targeted at your application then it may be worth spending CPU for minor mitigation. If you're getting 600Mbps+ being tossed at your hardware it may crush the machine and it will be pointless to do it there. The idea is to bounce the session or traffic off of your service somewhere enough to let the legitimate stuff through.

Thus why the suggestion to have your upstream provider can it when its frac on their multiple 100Gbps links before it gets to you.

Methanar
Sep 26, 2013

by the sex ghost


At this moment I have about 20gbps inbound altogether. Highest peak I've ever seen is about 32gbps, although this is growing very quickly. That's my baseline for legitimate traffic. Current CPU utilization is about 50% so maybe you're right, pushing everything through snort would cause everything to grind to a halt, or at least potentially start causing CPU wait.

I don't think upstream filtering is really what I want here. Some of my providers provide blackhole community strings, but ultimately I'd still need to write some sort of script to automatically detect an attack, who it's coming from and then advertise the blackhole string. And that's all assuming the source is well defined and something that's reasonable to block, and not some AWS /12 or ISP's /18 which could have legitimate users. Both of which I've seen garbage originate (or spoofed) from.

Sepist
Dec 26, 2005

FUCK BITCHES, ROUTE PACKETS

Gravy Boat 2k
In the interest of cheapness:

You could put a bump in the wire IPS inline on another server and use some kind of fail open device like a garland bypass switch to allow connectivity to continue if the IPS dies

ate shit on live tv
Feb 15, 2004

by Azathoth
For me since our normal traffic looks like DDoS (a shitload of short lived http/s transactions from mostly unique IPs), and we do around 70Gb/s over a global anycast network we just endure it. We've been hit a few times but with only like 10Gb/s which caused some slower then normal responses in a specific region but overall didn't affect us. The things that take us down are DNS provider attacks.

Otherwise just normal edge hardening. Control Plane Policing, ACL's that block everything except 80/443 to our specific service VIPs. We looked into Remote Trigger Blackhole so that we could save our other services, but unfortunately very few providers support it, and those that do only allow you to null route an entire /24, which is the exact opposite of what we want :/

And and yea, almost everything from the attacks we've gotten has been spoofed sources.

ragzilla
Sep 9, 2005
don't ask me, i only work here


ate poo poo on live tv posted:

For me since our normal traffic looks like DDoS (a shitload of short lived http/s transactions from mostly unique IPs), and we do around 70Gb/s over a global anycast network we just endure it. We've been hit a few times but with only like 10Gb/s which caused some slower then normal responses in a specific region but overall didn't affect us. The things that take us down are DNS provider attacks.

Otherwise just normal edge hardening. Control Plane Policing, ACL's that block everything except 80/443 to our specific service VIPs. We looked into Remote Trigger Blackhole so that we could save our other services, but unfortunately very few providers support it, and those that do only allow you to null route an entire /24, which is the exact opposite of what we want :/

And and yea, almost everything from the attacks we've gotten has been spoofed sources.

Who're your current transits? Pretty much everyone I transit with has dRTBH down to a /32.

doomisland
Oct 5, 2004

Trend is to do on host defenses and just scale that out. Servers can do line rate filtering at 10+Gbps via DPDK/eBFP/XDP. and there are a bunch of toolsets out there with that stuff. IPS and other crap for detection of less volumetric attacks. A big to do is to be able to get the appropriate metrics and then be able to product actionable events on the data for Ops folks to do. Just staring at a dashboard for DDoS detection doesn't work well.

Moey
Oct 22, 2010

I LIKE TO MOVE IT
Anyone running a virtualized firewall in production?

I am replacing a bunch of SRX240 next year, and am eyeballing the SRX345, but now wondering about the vSRX.

I have used the vSRX for test stuff, but no production.

Thanks Ants
May 21, 2004

#essereFerrari


Is there a historical reason for VLANs in IOS being in their own little file rather than just rolled into the switch config?

jwh
Jun 12, 2002

Thanks Ants posted:

Is there a historical reason for VLANs in IOS being in their own little file rather than just rolled into the switch config?

Yes, it's from CatOS having a vlan.dat.

At least, that was always my understanding.

Nowadays I don't know why they still do it.

Sepist
Dec 26, 2005

FUCK BITCHES, ROUTE PACKETS

Gravy Boat 2k
Ios-xe doesn't have it, and monolithic ios is basically dead now so you shouldn't really ever see it modern hardware

falz
Jan 29, 2005

01100110 01100001 01101100 01111010

Thanks Ants posted:

Is there a historical reason for VLANs in IOS being in their own little file rather than just rolled into the switch config?

It's because vtp is on. switch it to transparent and they'll show in config.

nescience
Jan 24, 2011

h'okay

Partycat posted:

Cucm/jabber contact stuff

Hey thanks, I'll play around with this more and try to use it, I was originally looking for a contact list .XML that could be imported client side, as in from the Cisco Jabber application itself, without access to IM&P server, but this sounds like a better way.

ate shit on live tv
Feb 15, 2004

by Azathoth

ragzilla posted:

Who're your current transits? Pretty much everyone I transit with has dRTBH down to a /32.

A little company that is looking to be bought out called Internap.

Kazinsal
Dec 13, 2011



Anyone at Live right now and planning on doing the NetApp FlexPod event?

ragzilla
Sep 9, 2005
don't ask me, i only work here


ate poo poo on live tv posted:

A little company that is looking to be bought out called Internap.

Ah, the ole virtual transit. It's kind of mind boggling they don't offer dRTBH since all it'd be is making arrangements with their transits in each POP, and translating a BGP community. They wouldn't even need to sink any traffic unless it sourced from another customer of theirs.

ate shit on live tv
Feb 15, 2004

by Azathoth

ragzilla posted:

Ah, the ole virtual transit. It's kind of mind boggling they don't offer dRTBH since all it'd be is making arrangements with their transits in each POP, and translating a BGP community. They wouldn't even need to sink any traffic unless it sourced from another customer of theirs.

It actually does work well for our anycast network though and the price is right. But yea, it's not ideal.

Partycat
Oct 25, 2004

I'm not at live this year (coworkers got to go), but if anyone here does collab and wants to network we can organize on IRC or twitter or something. I'm no Warcop but I don't think I'm completely inept.

psydude
Apr 1, 2008

Bit late to the DDoS show, but if your customer base is largely constrained to one geographic location (Midwest, Europe, whatever), you can shunt incoming requests outside of that region to a sorry page or another data center using DNS load balancing with geolocation profiling.

CrazyLittle
Sep 11, 2001





Clapping Larry
That does assume that your DDoS target is a DNS record and not the IP address itself

Moey
Oct 22, 2010

I LIKE TO MOVE IT

Partycat posted:

I'm no Warcop but I don't think I'm completely inept.

Can confirm, PartyCat ain't no slouch.

ate shit on live tv
Feb 15, 2004

by Azathoth
Is there a simple way to keep an OSPF adjacency from forming for a certain amount of time after it goes down? I'm sure you get crafty with EEM or event-options or something, but I'd rather just adjust a "hold-down" time or something like that.

The scenario is we have a pair of VPLS circuits, a backup and a primary that goes to all of our datacenters that we download a few gigs/sec over. We have BFD enabled over these things so that if we start dropping traffic, bfd will force the neighbor down, and the backup path will be utilized. This works fairly well, but I'd like to avoid a constant "flapping" situation where there is packetloss over one of the VPLS links, so BFD goes down, but then comes back up, brining up ospf, then bfd brings it down again, etc, etc.

Ideally if bfd detects a failure it should force the OSPF neighbor down for say 1minute, then let it attempt to negotiate up again. So whether the change is made with BFD, or OSPF, or whatever, I don't really care. Just as long as the flapping is borught down to a more useful level.

tortilla_chip
Jun 13, 2007

k-partite
ip event dampening

Judge Schnoopy
Nov 2, 2005

dont even TRY it, pal
I'm having trouble getting a remote site to communicate SNMP with my PRTG server.

The remote site has an ASA, inside interface 1 has a site-to-site VPN with my hub ASA. PRTG server is on the inside of the hub ASA. I can ping remote inside interface from the PRTG server.

I've told the remote ASA that inside1 is the management interface, enabled SNMP and syslog, gave it the PRTG server as the SNMP and Syslog host. No communication. I'm guessing the problem is that the ASA is not inspecting the outbound SNMP traffic and giving it to the VPN, but I have no idea what else I can do to make this happen. The ASA needs to send SNMP out inside1, receive it back on inside1, and get it inside the tunnel.

Any tips or hints on where I should be looking or anything I can double check?

ate shit on live tv
Feb 15, 2004

by Azathoth

tortilla_chip posted:

ip event dampening

That's only for interface flapping right? Plus it's cisco only. We are an Arista/Juniper shop.

tortilla_chip
Jun 13, 2007

k-partite
Assuming you want a solution specific to the VPLS links your best option is probably SLAX/python scripts on the boxes then. You could tweak the OSPF timers for throttling incoming LSAs, but that's global to the OSPF process and might have some unintended consequences with regard to convergence for other more stable links in your topology.

tortilla_chip fucked around with this message at 19:16 on Jun 30, 2017

Richard Noggin
Jun 6, 2005
Redneck By Default

Judge Schnoopy posted:

I've told the remote ASA that inside1 is the management interface

Just to be clear, you issued 'management-access inside1', correct?

edit: take a gander at this.

doomisland
Oct 5, 2004

Hello

https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20170629-snmp

Docjowles
Apr 9, 2009

It's cool there probably aren't like millions of Cisco devices out there allowing SNMP to their external interfaces with community string "public" or anything

...

...

Thanks Ants
May 21, 2004

#essereFerrari


Client has messed up massively, or let their architect go a bit crazy, so now they need some PoE switches with more than 8 ports for racks dotted around a building. Not usually a problem, except they want them to be completely silent so I need something fanless.

Are there any options other than the Catalyst 2960L-24PS? I don't really pay attention to fans in switches other than the airflow direction because normally they aren't being thrown in such weird places.

ate shit on live tv
Feb 15, 2004

by Azathoth

Thanks Ants posted:

Client has messed up massively, or let their architect go a bit crazy, so now they need some PoE switches with more than 8 ports for racks dotted around a building. Not usually a problem, except they want them to be completely silent so I need something fanless.

Are there any options other than the Catalyst 2960L-24PS? I don't really pay attention to fans in switches other than the airflow direction because normally they aren't being thrown in such weird places.

Juniper EX2200-C-12P-2G.

We use these for floor switches where we have more people then ports have been built out. Silent and do PoE, haven't had any issues with them since we installed them.

Moey
Oct 22, 2010

I LIKE TO MOVE IT

ate poo poo on live tv posted:

Juniper EX2200-C-12P-2G.

We use these for floor switches where we have more people then ports have been built out. Silent and do PoE, haven't had any issues with them since we installed them.

We use the 2200 line as well for some access switches, rock solid.

GreenNight
Feb 19, 2006
Turning the light on the darkest places, you and I know we got to face this now. We got to face this now.

Cisco Web Security authentication has been down since this morning, so my email has been blowing up. Good job Cisco, can't just make it pass through if authentication is down? gently caress.

psydude
Apr 1, 2008


Magical.

Adbot
ADBOT LOVES YOU

mythicknight
Jan 28, 2009

my thick night

I have a giant Cisco Unity voicemail box I want to get voicemails out of. Problem: the unity web portal sucks rear end, and I'm talking about 100+ voicemails I'd have to manually click on to save as etc.

Is there an easier way to export or extract these wavs from Unity? (currently it is not tied to any AD or Exchange infrastructure)

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply