Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
Myrridinos
Jan 7, 2010

ChubbyThePhat posted:

Keep me posted on if they give you any answers as to what went wrong here.

Sure

Adbot
ADBOT LOVES YOU

Aragaith
Jul 26, 2006

Kirisame Marisa.
In space.
Help.

The Macaroni posted:

Reading this makes me more than a little ill. Yikes. I've been through a couple of EMR downtime events (fortunately I don't support the EMR system at all) but never anything like a giant AD lockout. I would not want to be on that conference call.

Good news. We're mostly fixed aside from users who were not in today. Helpdesk can handle that.

The story I got was there was an attempt to import AD accounts into PeopleSoft. No one says what failed but it lead to 4500 accounts being disabled. When they re-enabled the accounts the passwords were somehow scrambled up. Thankfully there is an intranet password reset link, so we were assisting users with having a point of login to access this, enter their login ID, then it texts or emails(this still makes no sense as an option) a code to verify the user. Sadly about 40% of accounts did not have a phone number entered correctly on the backend, so some only had email as the option, which is locked out. We lack tools to flat out reset and unlock accounts because parent company compartmentalizes all functions and limits who has access to what. Helpdesk can help force password resets but they were getting swamped. If an account was locked from too many failed password attempts the intranet reset will fail with a generic error, which lead to Security Admins getting direct calls by us field techs, which also eventually got swamped. Eventually we got access to add phone numbers to peoples accounts, which kinda helped, but by the time we got it we were almost done everywhere.

Thankfully we are now back to functioning again.

Samizdata
May 14, 2007

Aragaith posted:

Good news. We're mostly fixed aside from users who were not in today. Helpdesk can handle that.

The story I got was there was an attempt to import AD accounts into PeopleSoft. No one says what failed but it lead to 4500 accounts being disabled. When they re-enabled the accounts the passwords were somehow scrambled up. Thankfully there is an intranet password reset link, so we were assisting users with having a point of login to access this, enter their login ID, then it texts or emails(this still makes no sense as an option) a code to verify the user. Sadly about 40% of accounts did not have a phone number entered correctly on the backend, so some only had email as the option, which is locked out. We lack tools to flat out reset and unlock accounts because parent company compartmentalizes all functions and limits who has access to what. Helpdesk can help force password resets but they were getting swamped. If an account was locked from too many failed password attempts the intranet reset will fail with a generic error, which lead to Security Admins getting direct calls by us field techs, which also eventually got swamped. Eventually we got access to add phone numbers to peoples accounts, which kinda helped, but by the time we got it we were almost done everywhere.

Thankfully we are now back to functioning again.

Well, looks like you are prepared for next time...

(You KNOW there will be a next time.)

Kurieg
Jul 19, 2012

RIP Lutri: 5/19/20-4/2/20
:blizz::gamefreak:
Anyone have a best guess as to what happened here?

Zil
Jun 4, 2011

Satanically Summoned Citrus


Kurieg posted:

Anyone have a best guess as to what happened here?


Spray painted spray foam that they stuck components in before it dried?

Partycat
Oct 25, 2004

“Art” I’d say and a dad level internet joke

Data Graham
Dec 28, 2009

📈📊🍪😋



Kurieg posted:

Anyone have a best guess as to what happened here?



http://spamusement.com/index.php/comics/view/6

Zamboni Apocalypse
Dec 29, 2009
Yeah, IIRC it's some game-specific case mod, Resident Evil maybe?

The Raspberry Jam Out Of Diskspace, by H. P. Lanparty.

Jaded Burnout
Jul 10, 2004


Zil posted:

Spray painted spray foam that they stuck components in before it dried?

Yeah I'm going to go with this. Although they probably put the components in and then sprayed it, or maybe even sprayed it into a normal old PC to burst it.

Marcade
Jun 11, 2006


Who are you to glizzy gobble El Vago's marshmussy?

Pickman's Modem, surely?

Zamboni Apocalypse
Dec 29, 2009

Marcade posted:

Pickman's Modem, surely?

"You fool, Warren is #$@^^/ NO CARRIER"

Arquinsiel
Jun 1, 2006

"There is no such thing as society. There are individual men and women, and there are families. And no government can do anything except through people, and people must look to themselves first."

God Bless Margaret Thatcher
God Bless England
RIP My Iron Lady
This?
https://www.youtube.com/watch?v=P7iW_VuR8d0

Marcade posted:

Pickman's Modem, surely?
The Dialup Handshake of Erich Zann?

mewse
May 2, 2006

I vaguely remember that being a photoshop from some satire article

Samizdata
May 14, 2007

Zamboni Apocalypse posted:

Yeah, IIRC it's some game-specific case mod, Resident Evil maybe?

The Raspberry Jam Out Of Diskspace, by H. P. Lanparty.

Marcade posted:

Pickman's Modem, surely?

Zamboni Apocalypse posted:

"You fool, Warren is #$@^^/ NO CARRIER"

Arquinsiel posted:

This?
https://www.youtube.com/watch?v=P7iW_VuR8d0

The Dialup Handshake of Erich Zann?

:golfclap:

Renegret
May 26, 2007

THANK YOU FOR CALLING HELP DOG, INC.

YOUR POSITION IN THE QUEUE IS *pbbbbbbbbbbbbbbbbt*


Cat Army Sworn Enemy
Hello

Does anybody here work for the 3rd party company I'm fighting with, I need to talk off the record about how loving stupid you are so I don't get fired.

Their proof that something's still broken is a copy/paste of an alarm that, in the far right column, says "cleared on 4/27"

Thom and the Heads
Oct 27, 2010

Farscape is actually pretty cool.

Renegret posted:

Hello

Does anybody here work for the 3rd party company I'm fighting with, I need to talk off the record about how loving stupid you are so I don't get fired.

Their proof that something's still broken is a copy/paste of an alarm that, in the far right column, says "cleared on 4/27"

could you please just do the needful on this request

Renegret
May 26, 2007

THANK YOU FOR CALLING HELP DOG, INC.

YOUR POSITION IN THE QUEUE IS *pbbbbbbbbbbbbbbbbt*


Cat Army Sworn Enemy

Thom and the Heads posted:

could you please just do the needful on this request

needful completed, please confirm on your end. And please don't confirm by looking at an alarm board, because I have one of those myself and I know how awful they can be.

In a brief moment of competence, I actually looked into the issue some and found that the access point that the customer is complaining is passing traffic just fine, but it's an incredibly small amount. I asked them to prove to me that it's actually down and not that, you know, no customers are actually using it. Still, it's remotely possible that something's wrong?

If I had it my way, I'd roll a tech and do a hard bounce just for the hell of it, but our field techs are being giant babies about dispatching to these devices, and until the procedure is finalized I have absolutely no power to tell them what to do. This company is dumb.

Renegret
May 26, 2007

THANK YOU FOR CALLING HELP DOG, INC.

YOUR POSITION IN THE QUEUE IS *pbbbbbbbbbbbbbbbbt*


Cat Army Sworn Enemy
But let's be real here

I'm only deep diving into this issue because it's good interview material. If I handled it the way my boss says to, I'd just say the modem's online, go gently caress yourself.

dogstile
May 1, 2012

fucking clocks
how do they work?
Hah, that reminds me of an issue i had recently where I had an "active connection" but the connection was maxing out at 20kbps

Renegret
May 26, 2007

THANK YOU FOR CALLING HELP DOG, INC.

YOUR POSITION IN THE QUEUE IS *pbbbbbbbbbbbbbbbbt*


Cat Army Sworn Enemy

dogstile posted:

Hah, that reminds me of an issue i had recently where I had an "active connection" but the connection was maxing out at 20kbps

Honestly I want to point my finger at the customer, but the truth is we're just as bad. The whole thing's a cluster gently caress, and it's all made worse by a gross lack of communication between our respective managements. We have project managers for both companies making decisions on how this is going to be managed, then not sending that information down to their respective support teams.

Fortune 500 to Fortune 500 communication and cross troubleshooting is the loving worst. We're responsible for the hardware, the customer's responsible for the software, except both the hard and software sucks rear end and neither side wants to admit fault. On top of all this, our field group is refusing to roll to troubleshoot these devices, so even when there's a legitimate issue there's nothing I can do.

I feel like I'm the only voice of reason in a sea of idiots.

Aunt Beth
Feb 24, 2006

Baby, you're ready!
Grimey Drawer

Renegret posted:

I feel like I'm the only voice of reason in a sea of idiots.
New thread title.

Renegret
May 26, 2007

THANK YOU FOR CALLING HELP DOG, INC.

YOUR POSITION IN THE QUEUE IS *pbbbbbbbbbbbbbbbbt*


Cat Army Sworn Enemy
Customer: Please provide an update.

Me: Well I see over the past 3 hours, the modem only passed about 200kb of traffic. That's unusually low but it's pretty normal for a modem that's just plugged in and not doing anything. Also, I can see your device is online, attached, has an ip, and is pingable. Is your device provisioned? Can you SSH into it?

Customer: Please provide an update.

Me: ...no update. *post angrily on the Something Is Awful message boards*

th vwls hv scpd
Jul 12, 2006

Developing Smarter Mechanics.
Since 1989.

Renegret posted:

Customer: Please provide an update.

Me: Well I see over the past 3 hours, the modem only passed about 200kb of traffic. That's unusually low but it's pretty normal for a modem that's just plugged in and not doing anything. Also, I can see your device is online, attached, has an ip, and is pingable. Is your device provisioned? Can you SSH into it?

Customer: Please provide an update.

Me: ...no update. *post angrily on the Something Is Awful message boards*

Keep providing the same update until someone with a pulse has the ticket escalated to them?

Proteus Jones
Feb 28, 2013



th vwls hv scpd posted:

Keep providing the same update until someone with a pulse has the ticket escalated to them?

Yeah, I'd just keep pasting the same "update" over and over.

Ain't no aggressive like passive-aggressive.

Renegret
May 26, 2007

THANK YOU FOR CALLING HELP DOG, INC.

YOUR POSITION IN THE QUEUE IS *pbbbbbbbbbbbbbbbbt*


Cat Army Sworn Enemy
I work for an ISP, I'm just going to ignore the update requests since it's not like people's opinions of my employer can get any lower anyway. This is service provider to service provider communication, so the bar is set so low that even Hitler can't limbo underneath it in his spot in hell.

Proteus Jones
Feb 28, 2013



Renegret posted:

I work for an ISP, I'm just going to ignore the update requests since it's not like people's opinions of my employer can get any lower anyway. This is service provider to service provider communication, so the bar is set so low that even Hitler can't limbo underneath it in his spot in hell.

Probably for the best. (but I'd still probably go the route of pasting the same info over and over because I'm vindictive and petty like that) (yes, my team loves me, why?)

Sprechensiesexy
Dec 26, 2010

by Jeffrey of YOSPOS

Renegret posted:

I work for an ISP, I'm just going to ignore the update requests since it's not like people's opinions of my employer can get any lower anyway. This is service provider to service provider communication, so the bar is set so low that even Hitler can't limbo underneath it in his spot in hell.

You either work for Centurylink or Telstra.

Or both and you are just ignoring emails you are sending to yourself.

ilkhan
Oct 7, 2004

I LOVE Musk and his pro-first-amendment ways. X is the future.
My office manager got one of those fake MS support phone calls, and listening to the office manager of an msp fake out this dumbass is hilarious.

"Facebook doesn't close. It's how I get to the interwebs."

And then they hung up on her.

ilkhan fucked around with this message at 00:17 on May 9, 2018

A Pinball Wizard
Mar 23, 2005

I know every trick, no freak's gonna beat my hands

College Slice
Anyone had any issues with running executables off network shares since the update to Windows 10 1803?

GreenNight
Feb 19, 2006
Turning the light on the darkest places, you and I know we got to face this now. We got to face this now.

A Pinball Wizard posted:

Anyone had any issues with running executables off network shares since the update to Windows 10 1803?

No. I setup 3 or 4 call center computers on 1803 this week and there are loads of programs I have to install off the network.

Myrridinos
Jan 7, 2010

ChubbyThePhat posted:

Keep me posted on if they give you any answers as to what went wrong here.

So I got my site stable and it is backing up properly now.

I am still not entirely sure what caused what, but there were a couple things wrong that could have caused it.

The backup software having 900GB incremental backups were most likely caused by enabling encryption in the backup software after the initial full or possibly the dynamic memory on the appliance virtual machine which is not supported due to certain linux databases being finicky about that.

The VHDX the Windows server was running on had problems as well. The vhdx was a dynamic expanding disk that had a filesystem that just barely exceeded the size of the physical drive it resided on. Severe file system fragmentation and a painful corrupting round of extensive raid controller crashes probably resulted in files being written to the non-existent storage near the end of the filesystem causing the virtual machine to go critical. I shrank the filesystem which had to be done using Linux, Windows refused due to immovable files, then created a new smaller fixed disk from the contents of the dynamic one.

The backup software has been re-deployed fresh by a senior engineer from Unitrends and I was given a full training for future deployments. They've had the best support I've seen from a vendor in a long while.

TITTIEKISSER69
Mar 19, 2005

SAVE THE BEES
PLANT MORE TREES
CLEAN THE SEAS
KISS TITTIESS




A Pinball Wizard posted:

Anyone had any issues with running executables off network shares since the update to Windows 10 1803?

Just spitballing, but is SMB1 in the picture somewhere?

kensei
Dec 27, 2007

He has come home, where he belongs. The Ancient Mariner returns to lead his first team to glory, forever and ever. Amen!


Anyone else going to Ignite? I'm kind of looking forward to it.

The Fool
Oct 16, 2003


My boss and I are. It’s my first time for a conference this big, should be fun.

Antioch
Apr 18, 2003

kensei posted:

Anyone else going to Ignite? I'm kind of looking forward to it.

I was but I quit the job that was going to foot the bill so I'm reasonably sure I'm not now.

Sywert of Thieves
Nov 7, 2005

The pirate code is really more of a guideline, than actual rules.

dogstile posted:

Hah, that reminds me of an issue i had recently where I had an "active connection" but the connection was maxing out at 20kbps

I remember when I was in university and somehow 'supported' my entire building (~300 students) and had a slow connection in my own room. With all that admin access to the network infrastructure, I escalated it about 4 times to higher and higher IT tech gurus trying to find the source of the problelm, before it occurred to me that it might be my own machine. After I rebooted it everything worked fine. :suicide:

Renegret
May 26, 2007

THANK YOU FOR CALLING HELP DOG, INC.

YOUR POSITION IN THE QUEUE IS *pbbbbbbbbbbbbbbbbt*


Cat Army Sworn Enemy

Merijn posted:

I remember when I was in university and somehow 'supported' my entire building (~300 students) and had a slow connection in my own room. With all that admin access to the network infrastructure, I escalated it about 4 times to higher and higher IT tech gurus trying to find the source of the problelm, before it occurred to me that it might be my own machine. After I rebooted it everything worked fine. :suicide:

Just last month, I realized I had been using a 100mbps home router for the past three years.

I get 300mbps service

I work in networking :negative:

Renegret fucked around with this message at 12:30 on May 9, 2018

A Pinball Wizard
Mar 23, 2005

I know every trick, no freak's gonna beat my hands

College Slice

TITTIEKISSER69 posted:

Just spitballing, but is SMB1 in the picture somewhere?

Yes. It is not my idea or fault and there is no way not to use it. But SMB1 is reinstalled on the affected machines.

The only part of the program causing problems is the one that opens a socket. The rest of the app is fine. I found an unsourced response to a stack overflow question saying MS disabled the ability to open sockets for executables on network shares, but obviously I can't present that to management. Worst of all, I have an 1803 VM and it's not having any problems at all.

I love my job generally but I can't escape the fact that the already old as gently caress tech we use is slowly being killed off completely while management does their best "this is fine" dog impression.

Bigass Moth
Mar 6, 2004

I joined the #RXT REVOLUTION.
:boom:
he knows...
I want to run some test VMs from my laptop. Can I install vmware work station and then esxi in that, and load the vms in there, or will workstation not allow that?

Adbot
ADBOT LOVES YOU

Rexxed
May 1, 2010

Dis is amazing!
I gotta try dis!

Bigass Moth posted:

I want to run some test VMs from my laptop. Can I install vmware work station and then esxi in that, and load the vms in there, or will workstation not allow that?

It works but you may have to enable their nested VT thing to get 64 bit guests in ESXi.
https://kb.vmware.com/s/article/2034803

  • 1
  • 2
  • 3
  • 4
  • 5