|
EoRaptor posted:Gpupdate /force has been our helpdesks go to first step for years. The other side of that is our group policies are so crazy it actually works a lot of the time to get things working again. That doesn't work when computer accounts are not ending up in the proper OUs though. 
|
#
?
Oct 7, 2019 22:31
|
|
|
|
| # ? Apr 24, 2024 07:34 |
|
|
EoRaptor posted:Gpupdate /force has been our helpdesks go to first step for years. The other side of that is our group policies are so crazy it actually works a lot of the time to get things working again. I had to call the helpdesk because my VPN client was broken, and I was working from home that day. Running GPUpdate /Force was the first thing he tried when I got him remotely connected to my work laptop. Y'know, the one at home. The one I was calling about not be able to connect to the VPN.
|
|
#
?
Oct 7, 2019 22:39
|
|
|
D. Ebdrup posted:I do wish people would replace 'I had to reboot to get x to work' with 'I had to mask the problems to get x to work, by rebooting', since that's effectively what they're doing.
|
|
#
?
Oct 7, 2019 22:50
|
|
|
Wizard of the Deep posted:I had to call the helpdesk because my VPN client was broken, and I was working from home that day. I worked a help desk position once where I was absolutely not permitted to deviate from certain set troubleshooting steps. Not saying it's necessarily what's happening here but having to document "I did this even though it was provably pointless in this scenario" is not all that unusual an exercise for low level techs. i hated it so much
|
|
#
?
Oct 7, 2019 22:55
|
|
|
Enola Gay-For-Pay posted:Honestly we have so many computers that are left on for a week with a dozen people logged in, half assed custom apps that leak memory everywhere, and people running five hundred suspicious browser windows at once that rebooting is the only solution to most of our dumb problems. I mean training and decent programmers would also work but we're not doing that so reboots it is. At this point I assume if a Windows machine has been running for more than 3 weeks and a specialty-market program is doing something uniquely weird it probably just needs a reboot. If the problem reoccurs after a reboot then I'll dig in to it, but if not then it's not worth the effort.
|
|
#
?
Oct 7, 2019 23:21
|
|
|
Wizard of the Deep posted:I had to call the helpdesk because my VPN client was broken, and I was working from home that day. This sounds familiar. I wonder if we work for the same company. I had a group of Windows provisioning folks at one office who were constantly whining that imaging a computer took hours upon hours to complete. We would ask them what they were doing differently from the runlist, and they'd say they were following the runlist to a T and it was still taking forever. Eventually, one of my teammates happened to be in that office, and he went to their lab and asked someone to start imaging a machine so he could observe. They were running Windows Update at the end of every build. This was not on the runlist. When my colleague pointed out that this was not on the runlist, they said, "Oh, we asked the IT director, who used to work helpdesk many moons ago, and he said to run Windows Update every time." "Why?" "Because?" "Why didn't you mention it was a step that wasn't on the runlist when we asked?" "Because we added it to our runlist!" The end result was a (temporary) tightening of training, disclaimers that, "THIS is the runlist, ENGINEERING makes changes, YOU do not," and having other directors tell former-helpdesk-director to get the gently caress out of our labs and stop trying to micromanage.
|
|
#
?
Oct 7, 2019 23:37
|
|
|
Dirt Road Junglist posted:This sounds familiar. I wonder if we work for the same company. We don't any more. My position was off-shored at the beginning of summer. Dirt Road Junglist posted:They were running Windows Update at the end of every build. This was not on the runlist. When my colleague pointed out that this was not on the runlist, they said, "Oh, we asked the IT director, who used to work helpdesk many moons ago, and he said to run Windows Update every time." Why was this taking so long? Either new deployments should be relatively up-to-date, which means there's little for WU to actually do, or you have a local copy of WSUS/SCCM/PatchingToolOfChoice pushing patches from the local network, which should be somewhat fast. Either way, please don't put unpatched devices out there unless there's a very good reason for it.
|
|
#
?
Oct 7, 2019 23:51
|
|
|
Wizard of the Deep posted:Why was this taking so long? Either new deployments should be relatively up-to-date, which means there's little for WU to actually do, or you have a local copy of WSUS/SCCM/PatchingToolOfChoice pushing patches from the local network, which should be somewhat fast. It was taking so long because they were mistakenly using an outdated image, plus their network is horrendous, and we don't use WSUS. All they had to do was leave the machine on the network long enough for PatchingToolOfChoice to shove the current baseline onto it. Which is what the runbook said to do. They were basically applying patches from WU at the same time as PatchingToolOfChoice.
|
|
#
?
Oct 8, 2019 00:12
|
|
|
We had an outage today due to the network connection to our dns servers going down, I thought of the thread koan.
|
|
#
?
Oct 8, 2019 00:18
|
|
|
Dirt Road Junglist posted:It was taking so long because they were mistakenly using an outdated image, plus their network is horrendous, and we don't use WSUS. All they had to do was leave the machine on the network long enough for PatchingToolOfChoice to shove the current baseline onto it. Which is what the runbook said to do. They were basically applying patches from WU at the same time as PatchingToolOfChoice. Gotcha. I haven't read the runbook either. In my defense, I don't have access to it 
|
|
#
?
Oct 8, 2019 01:27
|
|
|
Speaking of runbooks, my client's third party helpdesk must have a poo poo one that routes everything through security operations just to see if we can fix it first. If someone has a VPN issue it always comes to us "because the firewall?" despite the relevant entries being "Any, VPN collector IP address, <Cisco Anyconnect ports>" so that people can use it on the move without needing to . Ye know, like it's a VPN or something? It has to be in the runbook, because these tickets come to us at least once a week, and the response is always "send it to the dude who handles VPN accounts".
|
|
#
?
Oct 8, 2019 01:53
|
|
|
Wizard of the Deep posted:Gotcha. I haven't read the runbook either. In my defense, I don't have access to it I mean, our setup is janky af for a variety of reasons, so it's a valid call out. Just had to spend a chunk of my work day trying to figure out which ones of our WDS servers in the WDS OU were real and which were decommed without notification, and then found out our M&A onboarding team has recently split into net-new-M&A and old-M&A-we-didn't-migrate-correctly-a-the-time teams, so now I have twice as many people to track down and ensure they're using the correct runbooks. And honestly, all of this poo poo should be automated and totally turn-key. I don't get why everyone thinks doing poo poo manually is better.
|
|
#
?
Oct 8, 2019 01:59
|
|
|
Dirt Road Junglist posted:I mean, our setup is janky af for a variety of reasons, so it's a valid call out. Call a meeting with team leads and a tech rep from ALL your customers and take them through the run book each time it fucks up. Make them police each other to avoid the meetings.
|
|
#
?
Oct 8, 2019 03:37
|
|
|
If y'all wanna do my job, you're welcome to apply for it.
|
|
#
?
Oct 8, 2019 04:17
|
|
|
i barely even want to do my job.
|
|
#
?
Oct 8, 2019 04:36
|
|
|
Arquinsiel posted:Pfffft lookit this guy who gives a poo poo about RCA and doesn't have everyone just demanding that it be working already.
|
|
#
?
Oct 8, 2019 10:41
|
|
|
I managed to convince myself that WeWork deploy their networks to each location using automation and user-land switch ports are all templated etc. But the site suffering from a rogue DHCP server that I walked into yesterday would suggest that they do nothing like that at all.
|
|
#
?
Oct 8, 2019 13:24
|
|
|
D. Ebdrup posted:I'm sorry someone has hurt you this much. Literally every solution to things I can touch at the current client is "turn it off and turn it on again" so I can move onto the next thing to turn off and on again or advise a user to do it... Just leaving poo poo broke is also an option.
|
|
#
?
Oct 8, 2019 14:02
|
|
|
Thanks Ants posted:I managed to convince myself that WeWork deploy their networks to each location using automation and user-land switch ports are all templated etc. But the site suffering from a rogue DHCP server that I walked into yesterday would suggest that they do nothing like that at all. Oh god, don't tell me these things. I just got out of a meeting yesterday where our real estate guy told me we're about to add imaging infrastructure to a WeWork managed site. Every time I decide to stop drinking, something new comes up at work...
|
|
#
?
Oct 8, 2019 23:22
|
|
|
We had a dumb one today. A year or two ago, we had a client where we installed a new phone system and set up their network stuff for them when they moved into a new location. They're a local branch of a big company, so of course all their IT stuff is managed by head office. I think we cabled it and did the install, but all the equipment apart from the new phone system was stuff we don't administer. I remember having to call in to their IT people to make sure they could see stuff on their end when we racked their switches. The location was two buildings with a big parking lot between them, and ground was all solid concrete so rather than try to dig through that, they had us set up a point-to-point wireless link between the two buildings. All well and good so far. Last fall we got a subcontracted job from whoever manages their IT, to go look at a wireless AP they were having trouble with. I get there and the guys on site say it's working fine and aren't sure why I'm there since they got it working again already. While I was waiting in the queue for their IT support, I figured I'll trace the connections back just to make sure the AP is connected properly. That's when I discover that at they apparently aren't using the P2P wireless link anymore. They had some sort of problem with it at some point, getting new equipment to talk to their network over the link, and their head office IT couldn't figure it out, so rather than call us and ask about it they disconnected the radios and had someone string a bare cable up between the buildings. Just one ol' long cat6 cable strung up from a mast on one building up to the wall on the other, over the parking lot area. About 130 feet. I think it was at least shielded cable. But still. When I got through to their IT, I told them the AP is working and by the way do you know about this horrible not-to-code hack-job they've got the inter-building link running on? They "make a note of it." So last week we got another subcontracted job for an AP being down. One of our other IT techs got sent before I heard about it, so he got to rediscover all the gory details himself. His notes from the service order: quote:Upon arriving on site I noticed the wireless point to point hardware was removed and replaced with an outdoor ethernet cable. It was string up between two poles held with tie wraps at 90 degree angles. Cable was tie wrapped to a j-pole and wrapped around a telephone pole. Extreme stress on cable noticed given the distance. It was at this point that your IT rep asked for photos. Work was stopped following me sending photos. We got an email back today asking like "ok, so what time and materials do you need to fix it, will it require a lift or can it be done with ladders?" and we got send back a "lol gently caress off, none of this is remotely to code, we aren't fixing your hack job. We'll get you a quote to bury a cable or set up a radio link again.  "
|
|
#
?
Oct 9, 2019 00:12
|
|
|
Entropic posted:We had a dumb one today. Can't you run a high tension steel wire to appropriate anchors on both ends, then run an outdoor rated armored cat6/fiber cable and spiral wrap it to the high tension steel cable? I know I've seen it done before for longer overhead runs, but probably not that long. Stupid solutions to stupid problems!
|
|
#
?
Oct 9, 2019 00:22
|
|
|
Yes but the issue here is complying with building codes and other local regulations I reckon?
|
|
#
?
Oct 9, 2019 00:25
|
|
|
Methylethylaldehyde posted:Can't you run a high tension steel wire to appropriate anchors on both ends, then run an outdoor rated armored cat6/fiber cable and spiral wrap it to the high tension steel cable? I know I've seen it done before for longer overhead runs, but probably not that long. Stupid solutions to stupid problems! You probably could. Bear in mind the full scope of the job we were called in for was "have a technician come to site to site and call in to our IT line to help them troubleshoot why a wireless access point is not working." with a budget of like "2 hrs labour and maybe a new patch cable."
|
|
#
?
Oct 9, 2019 00:30
|
|
|
Sheep posted:Yes but the issue here is complying with building codes and other local regulations I reckon? Oh absolutely. I've just seen it done on permitted worksites so I know it's doable, but no idea what the limits are for something like that. Entropic posted:You probably could. "For the low low price of 20 hours, a manbasket rental, and 4 hours of licensed contractor time, we can bring your hilariously illegal cable job up to code. Orrrrr, we can set the wireless link back up. Your choice."
|
|
#
?
Oct 9, 2019 00:43
|
|
|
Methylethylaldehyde posted:Can't you run a high tension steel wire to appropriate anchors on both ends, then run an outdoor rated armored cat6/fiber cable and spiral wrap it to the high tension steel cable? I know I've seen it done before for longer overhead runs, but probably not that long. Stupid solutions to stupid problems! My understanding is you should never run copper network cable between buildings unless they share an electrical ground (e.g. a shed that gets its power from a house and doesn't have its own ground) since Weird poo poo can happen (e.g. potential difference) You also have to worry about lightning and so on if it's above ground. Far better to run fibre, underground, in a decent conduit.
|
|
#
?
Oct 9, 2019 09:24
|
|
|
You can get pretty much bulletproof p2p wifi equipment from Ubiquiti, even those rated for gigabit+ are not particularly expensive...
|
|
#
?
Oct 9, 2019 10:09
|
|
|
Wibla posted:You can get pretty much bulletproof p2p wifi equipment from Ubiquiti, even those rated for gigabit+ are not particularly expensive... I.e. exactly what they had originally but decided to rip down for some reason. 
|
|
#
?
Oct 9, 2019 19:45
|
|
|
Entropic posted:I.e. exactly what they had originally but decided to rip down for some reason. Caveman IT: GROG NOT UNDERSTAND THING, RIP THING OUT AND REPLACE WITH DUMBER, SIMPLER THING GROG UNDERSTAND It's basically that scene from Zoolander.
|
|
#
?
Oct 9, 2019 19:51
|
|
|
more like
|
|
#
?
Oct 9, 2019 21:52
|
|
|
In case any of you use a Mac and use iTerm2 - update ASAP. https://blog.mozilla.org/security/2019/10/09/iterm2-critical-issue-moss-audit/ tl;dr: Just *connecting to* an evil server can allow it run code on your machine
|
|
#
?
Oct 9, 2019 21:53
|
|
|
Ticket 1 came in:Ticket 1 posted:User: Our websites aren't working in the Citrix desktops, is something wrong? Ticket 2 came in a day later: Ticket 2 posted:User: The websites are down again, please check the Citrix desktops. I need a new career. These games are really starting to get under my skin.
|
|
#
?
Oct 10, 2019 15:25
|
|
|
Reply to ticket #3 with "see tickets #1 and #2" if you are feeling charitable. Reply to it with "no" if not.
|
|
#
?
Oct 10, 2019 15:42
|
|
|
hihifellow posted:Ticket 1 came in: I spent 3 hours on a conference call today investigating error messages in a newly updated system. Resolution: Custer Service Reps didn't read the god drat patch notes as they were instructed, error messages are by design to stop them from doing stupid poo poo they were previously getting away with.
|
|
#
?
Oct 10, 2019 15:55
|
|
|
Me: *emails affected groups, including our contact center* Subject: RESOLVED - <ISSUE> Body: <ISSUE> is now RESOLVED Contact Center Agent reply (serious): I just wanted to confirm that the issue is resolved? 
|
|
#
?
Oct 10, 2019 16:06
|
|
|
nexxai posted:Me: *emails affected groups, including our contact center* High level executive (serious): When are you sending the resolution notice?
|
|
#
?
Oct 10, 2019 16:08
|
|
|
CEO: JUST FIX IT NOW THIS IS AFFECTING PRODUCTVITY!!! sent from my iPhone 11XS
|
|
#
?
Oct 10, 2019 16:56
|
|
|
hihifellow posted:Ticket 1 came in:
|
|
#
?
Oct 10, 2019 17:13
|
|
|
|
|
#
?
Oct 10, 2019 18:29
|
|
|
nexxai posted:Me: *emails affected groups, including our contact center* Contact centers seem to attract the stupidest of the stupid. There was recently an email that went from our corporate IT that said, simply, "connect to the VPN for at least one consecutive hour at some time this week OR be connected in the physical office for one full hour. At the end of the week, reboot your computer" Many, many, MANY serious replies asking what to do. Or how to reboot.
|
|
#
?
Oct 10, 2019 18:39
|
|
|
|
| # ? Apr 24, 2024 07:34 |
|
|
The call center adjacent to my team's cube farm has giant TVs hanging from the ceiling for call/case stats and announcements (birthdays, work anniversaries, etc). They handed the reigns of the announcement slides over to a guy who is extremely from the internet, and they've been getting steadily more animated and elaborate. This month, it's clips from this monstrosity, overlaid with David S. Pumpkins backup dancers. https://www.youtube.com/watch?v=PUhuPn8_d0Q I'm lucky that it's not in my eye-line, but my teammates are quietly starting to lose their poo poo over it. Except none of us wants to make a fuss over it, because it's one of the few "fun" things the call center gets to do, and their lives suck enough as it is 
|
|
#
?
Oct 10, 2019 18:48
|
|