|
iospace posted:The fact that they set the starting max temp to -1 instead of you know, ZERO, makes me cringe. I have a good value overflow story! We had regular downtime for much of our infrastructure. We had a rolling set of reboots for every server/cluster/whatever every 48 days. With the size of our infrastructure, this meant that most days, something was down or in degraded mode or otherwise not working properly. Investigation revealed that we do this because the servers crash after 49 days or so. If we reboot them on day 48, all is well. If not, sometime during the next 24 hours, poof. This struck me as odd. That's a really weird and specific cadence. Why crash after 49ish days? What could be going on. What sort of regular, time-bound limit could we be running afoul of? My mind immediately went to "we're counting something, and it's overflowing." And since it's consistently crashing after a set time period, we had to be counting time. A signed int value is 4,294,967,295. 4,294,967,295 miliseconds is 49 days and some number of hours. I asked around, and sure enough, we had a process that was counting uptime in miliseconds. I had them check the code...and it was using a signed int value for that field. When it overflowed, it went haywire, rapidly consuming system resources until the whole system crashed. They made some code changes and no more crashes!
|
#
?
Jul 22, 2019 17:07
|
|
|
|
| # ? Apr 25, 2024 22:02 |
|
|
ConfusedUs posted:They made some code changes and no more crashes! 
|
|
#
?
Jul 22, 2019 17:12
|
|
|
AlexDeGruven posted:Same A particular NOC Guy posted:Stops telling you about any alerts...
|
|
#
?
Jul 22, 2019 17:13
|
|
|
Dirt Road Junglist posted:You're doing the lord's work, my dude. I think my greatest strength as a NOC person is to know when to shut the gently caress up. Said supervisor once got yelled on a conference call because he kept interrupting troubleshooting with useless nonsense and random bullshit that wasn't needed, all while talking in his long winded drawl that made it take 30 seconds to say "We have X number of customer complaints". He, of course, hasn't learned his lesson, and still gets on my case for not doing the exact same thing. Meanwhile, I'll sit there, take notes, and document tickets for the SMEs. Yet he thinks I'm a bad employee for not "driving troubleshooting" by not asking for updates every 30 minutes even though I'm listening to the conversation and know exactly what's going on. Yes there's times I need to get on people's asses to do their loving jobs, but for the most people don't like it when their systems are broken and know better than me on the best path to resolution. e: we probably work for the same place if you're one of those assholes who IM me directly for help instead of calling because they're afraid who will pick up the phone, while I'm trying to be lazy and slack off. Renegret fucked around with this message at 17:31 on Jul 22, 2019 |
|
#
?
Jul 22, 2019 17:25
|
|
|
Renegret posted:Cisco  e: The "49 days reboot" story is an oft-recurring refrain, I remember an era when entire enterprises would forcibly reboot to avoid 49-day uptime. In fact I'm pretty sure my current employer's 1-week VDI lifetime is a relic of it
|
|
#
?
Jul 22, 2019 17:40
|
|
|
I get the same alerts. I'd honestly prefer he never sent them to me anyway.
|
|
#
?
Jul 22, 2019 17:48
|
|
|
Renegret posted:I think my greatest strength as a NOC person is to know when to shut the gently caress up. Ugh, I had a supervisor like that in a previous sysadmin job. She was mad that I wasn't running around the building all day, asking people if they needed help. Um, does a fire fighter drive around town all day looking for smoke? No? So let me do my actual job, which is making sure all the poo poo you don't even know about continues working, like the janky file server that's also the AutoCAD licensing server that wants to trip over its own feet every three days. I have nothing but contempt for people who think they know how to do my job when I'm obviously competently handling my poo poo according to anyone with a working brain. Renegret posted:e: we probably work for the same place if you're one of those assholes who IM me directly for help instead of calling because they're afraid who will pick up the phone, while I'm trying to be lazy and slack off. Hah, I sincerely try not to do that. Unless it's one of the guys who's literally 15' to my right who I can yell, "HEY MIKE," at. I hate it when people IM me direct, so I try not to do it to anyone else. A fuckin' end user pinged my team's DL the other day with an email about a known issue, all, "HEY, I HAVE A PROBLEM, AND I'M TOO LAZY TO CALL TECH SUPPORT AND GET THE ESTABLISHED FIX FROM THEM. HOLD MY HAND, FOR I AM IMPORTANT." He got told to file a ticket.
|
|
#
?
Jul 22, 2019 18:11
|
|
|
Had a SQL server this morning randomly lose its SPNs. No idea why it happened, setspn -L <servername> listed them like normal, but absolutely nothing could connect to it using integrated security. Deleting and recreating them would let it work for about a minute before connections threw SSPI errors again. Nothing but successes in the security event logs (yes, that means everything gave a result of 0x0), even for events I know failed. Rebooting the server fixed it. The only thing we caught was reverse lookup for the IP went to a different hostname, which was fixed early on but resolved nothing until we had a window to reboot 2 hours later. Client would like an RCA and I got nothing ¯\_(ツ)_/¯
|
|
#
?
Jul 22, 2019 18:25
|
|
|
Dirt Road Junglist posted:Ugh, I had a supervisor like that in a previous sysadmin job. She was mad that I wasn't running around the building all day, asking people if they needed help. Um, does a fire fighter drive around town all day looking for smoke? No? So let me do my actual job, which is making sure all the poo poo you don't even know about continues working, like the janky file server that's also the AutoCAD licensing server that wants to trip over its own feet every three days. I can go on about that supervisor for days. I have a history with him. Micromanagers can loving drop dead for all I care. e: "I used to work your job I know how it is" dude, I was there, you spent so much time away from your desk running personal errands and taking personal phone calls that management had to assign an extra temp to your shift to make up the slack. I know, because I was the temp. Yet he tried to block my good review this year because of this single instance of me being 4 minutes late because of a car accident, and I didn't call in because I don't own a hands free device and I will not break the law and put myself and other motorists in danger by using a mobile device while driving (when convenient for me lmao) e: worst of all, he microwaves popcorn in the break room Renegret fucked around with this message at 19:12 on Jul 22, 2019 |
|
#
?
Jul 22, 2019 18:59
|
|
|
I wish ISPs would gently caress off with assigning port speed and duplex to <1Gbps links. As far as I can tell it's because some kit from the 90s didn't autonegotiate very well, and despite everything of that vintage being in a bin now, old habits die hard.
|
|
#
?
Jul 22, 2019 19:14
|
|
|
A Ticket Came In...quote:The internet is down. It turns out Open DNS decided to straight up drop all traffic from us. We were using them for filtering, but decided to switch providers because they were being a giant pain in the dick. We decided we could just have unfiltered DNS for a couple weeks or so before the new service began and just left everything pointed to OpenDNS ip's. Apparently they didn't like that and blocked ALL of our traffic just to spite us. I pointed my home computers to OpenDNS just to see if they were working, and yeah that confirmed they are giving us a giant middle finger.
|
|
#
?
Jul 22, 2019 19:35
|
|
|
A service acquired by Cisco turns to poo poo? Surely not
|
|
#
?
Jul 22, 2019 19:42
|
|
|
Data Graham posted:
my favorite Cisco story at this company. We had an major customer outage when a fan started dying and was spinning below spec. It was still spinning though, and the room was air conditioned well enough that the temperature was fine even without the fan, but the router went HOLY poo poo WE MUST PROTECT THE LINE CARDS and shut everything down. When the remote tech re-seated the fan, everything booted up just fine, but the router gave a warning about the fan and said it'd shut down in 60 seconds. In order to prevent another outage, the poor on site tech had to sit there and reseat the fan once a minute before the router shut down. There were no spare fans on site, so we had to have someone drive one to him from another location. In rush hour traffic. It took over 2 hours for the fan to get to him. Cisco wanted us to fix this by updating the firmware, right then and there, completely bypassing our change management and any sort of internal testing. And that's ignoring how the upgrade would've caused a service outage anyway during the reboot.
|
|
#
?
Jul 22, 2019 20:02
|
|
|
A text message came in. A picture of the "Network" screen of Windows Explorer with the "Network discovery is turned off" message. The text said "I'm in the physics room and I need help getting to the network." Having dealt with this user (the principal) for years, 90% of her issues revolve around her not remembering how to access network drives when there isn't a specific shortcut to them on her desktop. So it was pretty obvious what her issue was and what the solution was. I sent her a response. My response: "Please choose "This PC" instead of "Network" to view the network drives." 45 minutes later, I am called into the director's office to defend my "terse and unprofessional text message" and why I didn't "offer personal, face-to-face help." I pulled up my "end of summer, week before students return" ticket queue and showed it to him, with her ticket at the very bottom, and asked which he preferred, an efficient text message and personal responsibility, or a lengthy wait and twiddling thumbs. Then he lectured me for having too much to do. Millennials are killing the personal, face-to-face help industry.
|
|
#
?
Jul 22, 2019 20:32
|
|
|
Why are people texting you for support
|
|
#
?
Jul 22, 2019 20:46
|
|
|
Renegret posted:e: worst of all, he microwaves popcorn in the break room I was nodding along in solidarity until I got to this and immediately blurted out, "THAT MONSTER." Thanks Ants posted:Why are people texting you for support Why are people mad that you responded back in the same medium they used to contact you? And I also got a 1/5 star review for a similar thing (ticket came in, routine software request; sent the user the software and instructions and closed the case, as was procedure; got a bad review for not waiting for the user to let me know if it installed successfully (it did)), and my supervisor agreed with the user, even though I pulled up our software distribution doc and showed where it says, "Don't wait for the user to contact you back. If something goes wrong, they can file a troubleshooting ticket." Same supervisor, another bad review, because I gave them some boilerplate answer and the critical info was second in the list, not first. Like, come the gently caress on, if you're so hard up that you're bitching about having to read 2 sentences instead of 1, you're beyond my or anyone's help, and I wish you to die in a pit full of bed bugs. His response was to slow down and take more time personalizing my cases. Bitch, you want me to close 200% more cases than anyone else, but when I do things efficiently, you throw me under the bus? Nah. Luckily, that supervisor realized how much we all hated him and he left. Fuckin former Stream/Yahoo phone center supervisor...the worst.
|
|
#
?
Jul 22, 2019 20:53
|
|
|
Thanks Ants posted:Why are people texting you for support I'm the entire IT Department and this is the principal of the school. All administration enjoy the unmitigated pleasure of personally bothering me for minor issues instead of sending in tickets because administration and that's about it. It's not a hill I feel like dying on and we've been over numerous times why I can't exactly rock the boat or find a new job.
|
|
#
?
Jul 22, 2019 21:00
|
|
|
Agrikk posted:I think if you didn’t do this, or the other dumb or computer breaking things mentioned in this thread, you don’t belong in this thread. I recently sent a lot of random information from solarwinds to the new director of IT, some managers, my supervisor, and the networking team when I noticed a lot of extra traffic going out to our remote monitoring service's IP. It ended up being a bug where they turned on constant monitoring for holidays but forgot to turn it back off. Found because I was really curious why all that traffic was being generated for one IP and no one really knew what it was.
|
|
#
?
Jul 22, 2019 21:31
|
|
|
Thanks Ants posted:I wish ISPs would gently caress off with assigning port speed and duplex to <1Gbps links. As far as I can tell it's because some kit from the 90s didn't autonegotiate very well, and despite everything of that vintage being in a bin now, old habits die hard. Dirt Road Junglist posted:Luckily, that supervisor realized how much we all hated him and he left. Fuckin former Stream/Yahoo phone center supervisor...the worst.
|
|
#
?
Jul 23, 2019 01:08
|
|
|
A ticket came in, I sent it in two months ago pointing out to our ops team that a bunch of subdomains weren't pointing at the server clusters they needed to point to, and some were even going to the wrong data center entirely. Two weeks ago they asked if it was still a problem. They hadn't done a single thing, so now let me provide pages of data testing each and every subdomain I referenced, pointing out why and how it is wrong, and tie this ticket to the email thread with VPs on it asking why my deployment which is set to enable an entire subsidiary company to do their job is stuck as blocked. Again.
|
|
#
?
Jul 23, 2019 01:24
|
|
|
https://twitter.com/TomLawrenceTech/status/1153371714207830017?s=20
|
|
#
?
Jul 23, 2019 01:52
|
|
|
That’s amazing.
|
|
#
?
Jul 23, 2019 02:04
|
|
|
Time to inflict that on our deployment crew's slack channel tomorrow. 
|
|
#
?
Jul 23, 2019 02:21
|
|
|
I wish you to die in a pit full of bed bugs. [/quote] Oh god gently caress this thought even I'm freaking out just thinking about it. Bedbugs are a loving nightmare
|
|
#
?
Jul 23, 2019 04:51
|
|
|
RFC2324 posted:I wish you to die in a pit full of bed bugs. Oh god gently caress this thought even I'm freaking out just thinking about it. Bedbugs are a loving nightmare [/quote] DHCP-enabled bedbugs
|
|
#
?
Jul 23, 2019 05:27
|
|
|
Weatherman posted:DHCP-enabled bedbugs shrink your scope to a single IP with a stupid-long lease time and pray to your god that its male
|
|
#
?
Jul 23, 2019 06:24
|
|
|
I can't stop twitching now. all thing that might ever allow bedbugs to live must die in a fire
|
|
#
?
Jul 23, 2019 06:35
|
|
|
RFC2324 posted:I can't stop twitching now. all thing that might ever allow bedbugs to live must die in a fire Fire, specifically heating the entire apartment to high enough temps to denature a few key enzymes they need to live, is one of the few ways to kill bedbugs these days. They're resistant to almost every pesticide except for a few truly awful organo-phosphates, which are basically a name brand and a methyl group away from a big-boy nerve agent warcrime waiting to happen.
|
|
#
?
Jul 23, 2019 11:13
|
|
|
Methylethylaldehyde posted:Fire, specifically heating the entire apartment to high enough temps to denature a few key enzymes they need to live, is one of the few ways to kill bedbugs these days. They're resistant to almost every pesticide except for a few truly awful organo-phosphates, which are basically a name brand and a methyl group away from a big-boy nerve agent warcrime waiting to happen. I am quite aware, I've dealt with them. Thats where the pathological hatred comes from.
|
|
#
?
Jul 23, 2019 11:18
|
|
|
RFC2324 posted:I am quite aware, I've dealt with them. Thats where the pathological hatred comes from. Goondolences.
|
|
#
?
Jul 23, 2019 11:26
|
|
|
I have questions.
|
|
#
?
Jul 23, 2019 14:15
|
|
|
If it fits, I sits.
|
|
#
?
Jul 23, 2019 15:07
|
|
|
Arquinsiel posted:Stream as in Stream Global Services or some Yahoo product nobody cares about? The former. We picked up a lot of folks from Stream when I worked in the  call center. Most of the employees turned out to be solid to amazing, but the managers were horrible and none of them lasted very long in our corporate culture.RFC2324 posted:I am quite aware, I've dealt with them. Thats where the pathological hatred comes from.  That's why it's such an effective invective.I hope they're still in the walls of my old house, because the former landlady who still lives there deserves to be eaten to death in her sleep.
|
|
#
?
Jul 23, 2019 18:27
|
|
|
Dirt Road Junglist posted:The former. We picked up a lot of folks from Stream when I worked in the
|
|
#
?
Jul 23, 2019 22:30
|
|
|
Arquinsiel posted:I was on the Xbox contract with them in 2010 here in Ireland. Utterly poo poo management culture based purely on nepotism, employees ended up with genuine PTSD. Sounds about right, having also worked with a few former Xbox folks from there. No one who came over here was in their right mind. And at least one person actually had a mental break, split from his wife, and killed his pets after his contract expired. It was...not okay. Not ENTIRELY anyone's fault in this company (and my therapist keeps reminding me that other people aren't my responsibility), but he was one of my proteges, and I can't say the way he was treated by management didn't in some way contribute to his meltdown. I'm glad this place is nicer to our phone analysts, except when we've had either former IBM or former Stream folks who've tried to run it like a regular call center. One boss told all the contractors (it's a mix of contract and full time employees; contract-to-hire, for the most part) that their performance would be better if they didn't have any tchotchkes on their desks, because decorating your desk is a distraction. I was involved in a project to add gamification to our ticketing system so we could start tracking more metrics and add some "fun" ways to train people, and he immediately put the kibosh on it when he started. My partner on the project ended up leaving the company entirely as a result. gently caress that guy, but thankfully, he's long gone. I'm just worried we're gonna panic-hire another call center manager and go back to the dark ages again.
|
|
#
?
Jul 23, 2019 22:44
|
|
|
A ticket came in. "Setup Dual Monitors for $User". Directly assigned to me by another Service Desk tech. All right then, no big deal, even though $othertech has a bad habit of assigning poo poo to me with no notification or enough information in the ticket. Open the ticket and check the ticket details. "Setup Dual Monitors for $user". A completed task from $othertech with 10 minutes on it. God. Damnit. $othertech. So I email $othertech, "Hey, do you have more information on this ticket? Do they already have a monitor they need set up, do they need us to provide them a monitor, did a purchase order for monitors go through and they need to be deployed?" $othertech responds, "They have their own monitor. They need help setting up a second to dual monitor. Please feel free to contact the user for more details; the ticket has requested information." I check the ticket - ain't poo poo up ins. Nothing has changed.  I reply to $othertech, and CC $supervisor, "Who has the monitor that needs to be set up? I should not need to track down $user and ask them to fill me in on what they need when they have already contacted us with that information." $othertech responds, "They want to have $user helped to setups their second monitor as dual. Contact $user directly for more details, if you're confused and need more info - that's why we have email/phone available for all staff to communicate." Before I can reply and ask why the gently caress $othertech is being such a lazy poo poo and refusing to provide me the information I need to do the work, $supervisor chimes in: quote:I reviewed the ticket and it needs more information. "Setup Dual Monitor for $user" is a short description suitable for the ticket title, but the ticket details should explain what the user actually asked for or the problem they're encountering. We should not be telling other staff to contact the customer for more information; all the information required to complete the requested work should be in the ticket. It is the Service Desk's job to get all of the necessary information from the customer and add it to the ticket when handling a call/email. 
|
|
#
?
Jul 23, 2019 23:01
|
|
|
dragonshardz posted:A ticket came in. "Setup Dual Monitors for $User". Directly assigned to me by another Service Desk tech. All right then, no big deal, even though $othertech has a bad habit of assigning poo poo to me with no notification or enough information in the ticket. Hnnnnnggggggggg
|
|
#
?
Jul 23, 2019 23:29
|
|
|
Dirt Road Junglist posted:Sounds about right, having also worked with a few former Xbox folks from there. Oh man, I bet that felt great. Sometime a week or so ago I sent the third party helpdesk a mail to raise a ticket with another third pary that does O365 poo poo to change all the distributionlist@clientcorp.onmicrosoft.com addresses to distributionlist@clientdivision.com addresses so as to avoid bypassing FireEye ETP. Since then I have been replying to "use cannot email @clientcorp.onmicrosoft.com address, plz whitelist!  " tickets with that email and "see attachment". Today I got to send it back to the very same tech I first sent it to 
|
|
#
?
Jul 23, 2019 23:42
|
|
|
....if you’re using a third party email filter provider why isn’t the tenant locked down to only allow email coming from the filtering provider? The only way in or out of our tenant (or on-prem smtp relay) is through proofpoint.
|
|
#
?
Jul 24, 2019 00:03
|
|
|
|
| # ? Apr 25, 2024 22:02 |
|
|
Arquinsiel posted:You're in the Beaverton area then I guess? That office had a TB outbreak... It felt loving awesome. I really, really don't like having a ticket assigned to me in the blind and doubly so when there's no details on what the previous tech did. Document loving everything or so help me... If I have to reassign a ticket to someone, I make dead sure there's enough context somewhere in the ticket for them to actually do poo poo, and if they ask for more information I don't tell them to gently caress off and contact the user themselves. That way lies madness.
|
|
#
?
Jul 24, 2019 00:14
|
|