Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
The only HA I ever worked with that worked as designed was a VRRP setup with a couple mikrotik CCR’s. Even then the failover was a script and it was highly complicated.

I suspect the reason behind this one is as you stated. The Enterasys S1 the next hop up from the checkpoint and/or the the core S4 behind it didn’t want to see another MAC address.

Regardless, I’m not happy about the fact that the simple act of blocking an application caused a six figure firewall solution to poo poo itself.

Adbot
ADBOT LOVES YOU

devmd01
Mar 7, 2006

Elektronik
Supersonik

Farking Bastage posted:

I can tell you that the expensive firewalls have quirks of their own. 15 minutes before quitting time today, something went wonky with our *VERY* expensive Checkpoint 15000 HA pair. We pushed an application rule to block snapchat and something glitched. The primary poo poo itself completely, but not in such a way that the HA backup would sense it and take over. Forced a failover to the secondary, but it still didn't want to work right. We ended up having to revert the config on the primary, fail it back over then revert the secondary. I'll be on the phone with someone in Israel a lot tomorrow looks like :(

That sounds about right for Checkpoint!

Methanar
Sep 26, 2013

by the sex ghost
We spent more than twice the value of my house on bullshit machine learning blackbox ddos mitigation devices that I currently have in passive mode doing nothing because I don't trust them to not be invisibly loving up everything.

Buying these things were almost certainly the worst mistakes we made when building out a datacenter. We could have purchased 50gbps of bandwidth for 5 years for what we spent on these loving things.

Several times these things have caused horrible horrible impossible to troubleshoot cascading failures. For example, something goes wrong so logging spikes. The spike in logging triggers some ML horseshit to start invisibly dropping traffic to or from AWS (because thats where our logging services were hosted) because this spike of traffic is an anomaly. Same for DNS

What happens when logging traffic or DNS gets dropped? Obviously you send more to log the failure to log or request DNS again.

Never again.

Methanar fucked around with this message at 04:00 on Jan 9, 2018

sfwarlock
Aug 11, 2007

Zamboni Apocalypse posted:

I believe the correct version for incoming fire is "For what we are about to receive, dear Lord make us thankful."

I always use "... may we be truly grateful."

Judge Schnoopy
Nov 2, 2005

dont even TRY it, pal

Methanar posted:

We spent more than twice the value of my house on bullshit machine learning blackbox ddos mitigation devices that I currently have in passive mode doing nothing because I don't trust them to not be invisibly loving up everything.

Buying these things were almost certainly the worst mistakes we made when building out a datacenter. We could have purchased 50gbps of bandwidth for 5 years for what we spent on these loving things.

Several times these things have caused horrible horrible impossible to troubleshoot cascading failures. For example, something goes wrong so logging spikes. The spike in logging triggers some ML horseshit to start invisibly dropping traffic to or from AWS (because thats where our logging services were hosted) because this spike of traffic is an anomaly. Same for DNS

What happens when logging traffic or DNS gets dropped? Obviously you send more to log the failure to log or request DNS again.

Never again.

Could you lower the sensitivity so it would take a disaster-level ddos to trigger action?

Or whitelist your cloud service IPs and DNS forwarders so it doesn't interfere with business traffic?

Methanar
Sep 26, 2013

by the sex ghost

Judge Schnoopy posted:

Could you lower the sensitivity so it would take a disaster-level ddos to trigger action?

Or whitelist your cloud service IPs and DNS forwarders so it doesn't interfere with business traffic?

No*

tldr
A huge amount of our traffic is UDP and my WAN traffic is very asymmetric. There is zero way that I can enforce that traffic exits through the same ddos appliance that it enters. This means it is impossible for these things to have any meaningful view of what UDP traffic is actually doing, and TCP traffic insight is reduced. Also unless I whitelist all of amazon's blocks which is pretty close to removing the device from service entirely like I have right now, I can never guarantee that IPs are going to be within a given whitelist. DNS whitelisting isn't a thing. The way the thresholds are generated, if an IP suddenly moves it will almost certainly immediately be flagged as anomalous and almost certainly blocked until a human intervenes because log transmission is continous and will look like an attack.

Ultra simplified view

wan1 -> ddos1 -> network |
wan2 -> ddos2 -> network |


The real answer to ddos mitigation is don't even remotely try to do it yourself. Properly harden your nginx or haproxy instances to flush their connection tables as necessary to avoid the low hanging fruit. If anything volumetric happens, pray that you chose a provider that supports bgp community strings to blackhole traffic then use that and the source is something that is reasonable to blackhole, not the entirety of comcast or something. Anything more serious of a problem or larger scale than that, you need to do bigger things like have a geographically distributed presence and do fancy things with BGP any casting and using sacrificial sites

Methanar fucked around with this message at 06:20 on Jan 9, 2018

Methylethylaldehyde
Oct 23, 2004

BAKA BAKA

Methanar posted:

Anything more serious of a problem or larger scale than that, you need to do bigger things like have a geographically distributed presence and do fancy things with BGP any casting and using sacrificial sites

Man, the cyberpunk dystopia is here, only instead of hot half-cyborg Asian chicks and seedy Japanese diners, it's some underpaid nerd in an office trying to cope with 100,000 infected webcams sending billions of dickpicks per second and crashing the servers because we as a race can't have nice things.

Steakandchips
Apr 30, 2009

Methylethylaldehyde posted:

Man, the cyberpunk dystopia is here, only instead of hot half-cyborg Asian chicks and seedy Japanese diners, it's some underpaid nerd in an office trying to cope with 100,000 infected webcams sending billions of dickpicks per second and crashing the servers because we as a race can't have nice things.

Beautifully put.

Neddy Seagoon
Oct 12, 2012

"Hi Everybody!"

Zamboni Apocalypse posted:

I believe the correct version for incoming fire is "For what we are about to receive, dear Lord make us thankful."

Sorry, no, this is the wrong prayer as they're the one sending out the patch over their network..

For something being caused by their own actions they want Shepard's Prayer; "Please God, don't let me gently caress this up".

Proteus Jones
Feb 28, 2013



Bob Morales posted:

After about two trips in the cold and snow I bought a UPS with a web interface for the power outlets :effort:

Yeah, our standard rack buildout now includes 2 non-negotiable devices. A multiport serial console server and a rack PDU.

Because it's been historically shown that we can't trust on-site people to find their rear end with both hands and a map (and honestly, dealing with that stuff isn't their job). Also, gently caress sending one of T1/2 guys 50 miles to cycle power.

Thanks Ants
May 21, 2004

#essereFerrari


Do you drop in a cheap DSL circuit for OOB access to the serial server?

The Muffinlord
Mar 3, 2007

newbid stupie?

Methylethylaldehyde posted:

Man, the cyberpunk dystopia is here, only instead of hot half-cyborg Asian chicks and seedy Japanese diners, it's some underpaid nerd in an office trying to cope with 100,000 infected webcams sending billions of dickpicks per second and crashing the servers because we as a race can't have nice things.

I'm going to print this and hang this on my cube wall.

For content, we just rolled out our new USB mass storage lockdown, mandated and designed by the nationwide corporation that is our parent organization only when they want to tell us to do stuff or demand that the hospital make more money. This is fine and dandy because our doctors are all morons and nobody wants to do infosec training for nurses who already don't have the patience to deal with these computer modems, except that the new filters prevent us from installing any USB hardware at all, not just Dr. Iknowbetter's infected SanDisk.

Thankfully they included a convenient way around it for those of us with AD access but I really wish the networking team and corporate would stop "fixing" problems by offloading them to the helpdesk. I also wish I'd have won the drat lottery. And a pony.

Proteus Jones
Feb 28, 2013



Thanks Ants posted:

Do you drop in a cheap DSL circuit for OOB access to the serial server?

Nah, that's on whichever group (both internal and external customers) actually runs that datacenter. The stuff we manage on our single rack is behind all that.

E: sorry misunderstood what you were asking first.

Proteus Jones fucked around with this message at 16:43 on Jan 9, 2018

MF_James
May 8, 2008
I CANNOT HANDLE BEING CALLED OUT ON MY DUMBASS OPINIONS ABOUT ANTI-VIRUS AND SECURITY. I REALLY LIKE TO THINK THAT I KNOW THINGS HERE

INSTEAD I AM GOING TO WHINE ABOUT IT IN OTHER THREADS SO MY OPINION CAN FEEL VALIDATED IN AN ECHO CHAMBER I LIKE

Thanks Ants posted:

My experience with Sonicwall has been terrible. We got onto the Gen6 train far too early (had no option though unless we wanted to buy old hardware) and it was a complete shitshow for a very long time. I still would try and avoid having things like a VLAN-tagged WAN interface because so much stuff just flat-out broke the last time I tried it.

I have a Fortigate E-series running 5.6 and it's nice but the UI tries to be too friendly. The amount of poo poo you can't edit once it's in use is infuriating as well, and the CLI is pretty horrific.

In short: all SMB UTM type appliances are pieces of poo poo in their own ways.

Their CLi is A LOT better than it used to be, we haven't run 5.6 code yet, still on 5.4 (some devices on 5.2), but it's getting better, the issue is the stark lack of documentation on poo poo. Every time I call support I make sure to log the SSH/CLi sessions they use because there's always something new that I didn't know about.

Proteus Jones posted:

Yeah, our standard rack buildout now includes 2 non-negotiable devices. A multiport serial console server and a rack PDU.

Because it's been historically shown that we can't trust on-site people to find their rear end with both hands and a map (and honestly, dealing with that stuff isn't their job). Also, gently caress sending one of T1/2 guys 50 miles to cycle power.

Those serial console servers are loving awesome, we have a digibox at one of our larger customers data centers where we manage an HA pair of firewalls and 2x8 stacks of cisco switches and it's awesome, I've only needed it twice, but it was a god send when I did. That customer is also not in the contiguous 48, so it really would have been annoying (and awesome) to have to fly there to fix a thing.

MF_James fucked around with this message at 17:19 on Jan 9, 2018

null_pointer
Nov 9, 2004

Center in, pull back. Stop. Track 45 right. Stop. Center and stop.

Methylethylaldehyde posted:

Man, the cyberpunk dystopia is here, only instead of hot half-cyborg Asian chicks and seedy Japanese diners, it's some underpaid nerd in an office trying to cope with 100,000 infected webcams sending billions of dickpicks per second and crashing the servers because we as a race can't have nice things.

Goin' into the Funny Forum Quotes thread. God drat.

RedMagus
Nov 16, 2005

Male....Female...what does it matter? Power is beautiful, and I've got the power!
Grimey Drawer
How much will I hate my life if I try to learn Salesforce? They're looking for someone to use it as a Customer Resource Management tool, run reports for invoices and track projects. It seems like a trap, but not sure how much worse it would be compared to helldesk.

chin up everything sucks
Jan 29, 2012

RedMagus posted:

How much will I hate my life if I try to learn Salesforce? They're looking for someone to use it as a Customer Resource Management tool, run reports for invoices and track projects. It seems like a trap, but not sure how much worse it would be compared to helldesk.

Learn to code for salesforce instead of being a service monkey. IT to BizApps is a decent move, but IT to report-runner is going to kill you.

bitterandtwisted
Sep 4, 2006




It's been a while since I've had a user who I dread seeing tickets from, but we've got a good one now. It's like tech-supporting my granny.
Today a colleague was troubleshooting why her built-in webcam was't working. After about an hour on a remotes session, he figured it out. The laptop lid was closed.

She's a department head for a science based company and can't be older than 40.

iospace
Jan 19, 2038


bitterandtwisted posted:

It's been a while since I've had a user who I dread seeing tickets from, but we've got a good one now. It's like tech-supporting my granny.
Today a colleague was troubleshooting why her built-in webcam was't working. After about an hour on a remotes session, he figured it out. The laptop lid was closed.

She's a department head for a science based company and can't be older than 40.

This is very common with doctors. They learn so much in their field how to do anything else slides right out.

mattfl
Aug 27, 2004

iospace posted:

This is very common with doctors. They learn so much in their field how to do anything else slides right out.

Can confirm, I work in a hospital.

Ursine Catastrophe
Nov 9, 2009

It's a lovely morning in the void and you are a horrible lady-in-waiting.



don't ask how i know

Dinosaur Gum

bitterandtwisted posted:

It's been a while since I've had a user who I dread seeing tickets from, but we've got a good one now. It's like tech-supporting my granny.
Today a colleague was troubleshooting why her built-in webcam was't working. After about an hour on a remotes session, he figured it out. The laptop lid was closed.

She's a department head for a science based company and can't be older than 40.


In her defense, if it was docked (it sounds like it was) she may have just thought the monitor had a webcam like a lot of all-in-ones/macs do v:shobon:v

Sickening
Jul 16, 2007

Black summer was the best summer.
FYI, have really bad issues with kb4057247 on some dell laptops. My desktop guys are having a hard time nailing down the exact problem. Just FYI.

Knormal
Nov 11, 2001

The Muffinlord posted:

For content, we just rolled out our new USB mass storage lockdown, mandated and designed by the nationwide corporation that is our parent organization only when they want to tell us to do stuff or demand that the hospital make more money. This is fine and dandy because our doctors are all morons and nobody wants to do infosec training for nurses who already don't have the patience to deal with these computer modems, except that the new filters prevent us from installing any USB hardware at all, not just Dr. Iknowbetter's infected SanDisk.

Thankfully they included a convenient way around it for those of us with AD access but I really wish the networking team and corporate would stop "fixing" problems by offloading them to the helpdesk. I also wish I'd have won the drat lottery. And a pony.
When our Group Policy people, who I've mentioned a few times in this thread before, tried to turn off USB storage at our place a few years ago they accidently turned off the USB ports entirely. We figured this out when everyone started calling the helpdesk at the same time because their mouse stopped working.

Luckily our being behind the times worked in our favor in this case because most people were still on PS/2 keyboards.

A Pinball Wizard
Mar 23, 2005

I know every trick, no freak's gonna beat my hands

College Slice
Why the gently caress did we think it was a good idea to make our own email client? :smithicide:

Thanks Ants
May 21, 2004

#essereFerrari


:stare:

MF_James
May 8, 2008
I CANNOT HANDLE BEING CALLED OUT ON MY DUMBASS OPINIONS ABOUT ANTI-VIRUS AND SECURITY. I REALLY LIKE TO THINK THAT I KNOW THINGS HERE

INSTEAD I AM GOING TO WHINE ABOUT IT IN OTHER THREADS SO MY OPINION CAN FEEL VALIDATED IN AN ECHO CHAMBER I LIKE

A Pinball Wizard posted:

Why the gently caress did we think it was a good idea to make our own email client? :smithicide:



ilkhan
Oct 7, 2004

I LOVE Musk and his pro-first-amendment ways. X is the future.

A Pinball Wizard posted:

Why the gently caress did we think it was a good idea to make our own email client? :smithicide:
I give up, what's the punch line?

incoherent
Apr 24, 2004

01010100011010000111001
00110100101101100011011
000110010101110010
:pray: please tell me it also clacks as you enter your email in like a old timey typewriter and the sound is hardcoded in VB6 :pray:

Thanks Ants
May 21, 2004

#essereFerrari


Lots of AOL sounds built in

Sirotan
Oct 17, 2006

Sirotan is a seal.


A Pinball Wizard posted:

Why the gently caress did we think it was a good idea to make our own email client? :smithicide:

Do you work for Nomx?

A Pinball Wizard
Mar 23, 2005

I know every trick, no freak's gonna beat my hands

College Slice

incoherent posted:

:pray: please tell me it also clacks as you enter your email in like a old timey typewriter and the sound is hardcoded in VB6 :pray:

Of course not! Most of it was rewritten in .net just last year!

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
Centurylink handed off the /25 belonging to one of my remotest of remote sites elsewhere today. Took about 6 hours of talking to people to get it sorted. :suicide:

Bunni-kat
May 25, 2010

Service Desk B-b-bunny...
How can-ca-caaaaan I
help-p-p-p you?

A Pinball Wizard posted:

Why the gently caress did we think it was a good idea to make our own email client? :smithicide:

I’m sorry but

Ahahahahahhahahahahahahaaaaaaaaa...


Sickening posted:

FYI, have really bad issues with kb4057247 on some dell laptops. My desktop guys are having a hard time nailing down the exact problem. Just FYI.

We’re an all Dell institution, what kind of errors are you seeing? It’d be nice to get ahead of that.

incoherent
Apr 24, 2004

01010100011010000111001
00110100101101100011011
000110010101110010

A Pinball Wizard posted:

Of course not! Most of it was rewritten in .net just last year!

One of the joys in my life is finding boutique software written in a blend of vb6\.net\foxpro.

GreenNight
Feb 19, 2006
Turning the light on the darkest places, you and I know we got to face this now. We got to face this now.

Those of you with Dell laptops, ever seen an issue where the trackpad/mouse freezes every 5 seconds for a second or two constantly? Working on a laptop with that issue and I can't figure it out. Tried a new hard drive, formatted. In Windows 10.

Bunni-kat
May 25, 2010

Service Desk B-b-bunny...
How can-ca-caaaaan I
help-p-p-p you?
Check your sata cables for April Fools.

RFC2324
Jun 7, 2012

http 418

GreenNight posted:

Those of you with Dell laptops, ever seen an issue where the trackpad/mouse freezes every 5 seconds for a second or two constantly? Working on a laptop with that issue and I can't figure it out. Tried a new hard drive, formatted. In Windows 10.

i have this issue with the track pad on my precision m6600, but it works fine with a mouse. never did figure out the issue, and it has persisted through multiple os installs and reseating the cable.

let me know if you figure it out

GreenNight
Feb 19, 2006
Turning the light on the darkest places, you and I know we got to face this now. We got to face this now.

It happens with a mouse too. Frustrating.

Aunt Beth
Feb 24, 2006

Baby, you're ready!
Grimey Drawer

A Pinball Wizard posted:

Of course not! Most of it was rewritten in .net just last year!
I was going to guess you worked for IBM and were talking about Notes until you said it was rewritten. IBM is never dragging that corpse out of Eclipse/Java.

Adbot
ADBOT LOVES YOU

dragonshardz
May 2, 2017

GreenNight posted:

Those of you with Dell laptops, ever seen an issue where the trackpad/mouse freezes every 5 seconds for a second or two constantly? Working on a laptop with that issue and I can't figure it out. Tried a new hard drive, formatted. In Windows 10.

I have a fix for this! The TL;DR is the Windows PTP drivers are poo poo and every time you touch the pad, they wait for you to make a gesture. You can replace them with the Synaptics drivers and the problem disappears.

  • 1
  • 2
  • 3
  • 4
  • 5