|
jre posted:3 hours later, hosting company still apparently unable to locate on button for server they switched off by mistake. Next time you buy mission critical hosting make sure your server has ipmi/kvm access. Of course that doesn't help when the colo customer below you comes in and rips all the cables out of your servers, removes them from the racks and just tosses them on the floor because "it was easier for me to work on my server."
|
#
¿
Aug 15, 2013 01:27
|
|
|
|
| # ¿ Apr 26, 2024 03:46 |
|
|
dennyk posted:Are there seriously data centers who allow colo customers to have unsecured hands-on access to other customers' systems these days? Yes it really happened. It was a couple of years ago so beer has kind of hazed up the memories but in a nutshell it was a shitstorm of fuckups. I had gone out to exercise leaving my phone inaccessible on what I recall was a Monday evening. Meanwhile halfway across the country a NOC tech escorts some graybeard to his sever collocated below our set of 5 production servers. I don't recall exactly what the grey beard was up to but I guess he was being slow and difficult. During this the NOC tech gets pulled away to help put out some other fire (not literally.) I guess at this point the grey beard gets frustrated, rips our cabling out and pops the severs out of the rack, setting them aside while he continues to align the ions on his ram or whatever the gently caress he felt was so important. 2 hours later and back on my side of the planet I reach down for my phone to be greeted with a wall of frantic texts saying almost everything is down. I get in touch with the data center and they tell me the automatic monitoring for my servers had been disabled when we set them up (this is another story all together) and they'd need to get a NOC tech over to that rack to see what's going on. I hear back shortly afterwords saying our servers had been de-racked and they'd need some time to get them back on and figure out what the gently caress happened. Everything came back not too much later but we did have to rebuild one raid array. I wound up getting the full story from an associate of mine who works there later that week when I followed up for our SLA credit. It could have been oh so much worse but luckily it was a Monday after business hours and none of our hardware was permanently damaged. Sorry if there are any horrible typos in this, it's been typed on my phone.
|
|
#
¿
Aug 15, 2013 03:46
|
|
|
Started a new job today, I think I'm in over my head. I'm managing about 50 users with 2 workstations each, plus a laptop, their phones and a 140 server render farm/computational cluster. Active directory is on server 2k3 and LDAP is managed on a sun box from 1999. Backups are done on tape and everyone has write access to everything. Plus side is my boss is pretty nice. God help us if anyone opens crypto locker though.
|
|
#
¿
Dec 24, 2013 01:43
|
|