|
Configured a couple of interfaces on a firewall today remotely in advance of that box getting moved to our site (I'm not doing the work this weekend hence getting things prepped in advance). Was connected via a VPN, added the public IP address of our connection to a spare interface and....got kicked out. I guess it got confused with the idea of a VPN client being connected from an internet address that matched one of its own interfaces, should have seen that coming really. Had to hotspot off my phone to finish the job.
|
#
?
Apr 24, 2019 19:03
|
|
|
|
| # ? Apr 19, 2024 08:25 |
|
|
hihifellow posted:An emergency ticket came in, a client's only Citrix server powered off in the middle of the day and now nobody can work! quote:Boot it back up, check the event logs and find the shutdown event. quote:Started by the person who filled the ticket.
|
|
#
?
Apr 24, 2019 20:05
|
|
|
That’s why you have one domain admin account that everyone shares so who knows, who broke it.
|
|
#
?
Apr 24, 2019 20:07
|
|
|
A ticket came in! We just lost all network infrastructure in one of our buildings! My boss and I head over to the network closest that has all the fiber that feeds the rest of the network closets in the building to see our 3rd party wiring guy in there patching down cables. All our of switches are off and the PDU that feeds said switches are off. We ask the dude wtf he's doing and he's like uhhh nothing, just patching cables. We notice the main UPS in the rack is sitting on the floor having been remove from said rack. Apparently it was "in his way" and he unscrewed it from the rack and pushed it out of the rack a little. Doing so caused it to hit the power button on the floor because of how it was angled and put it in some sort of bypass mode, which, while it was still powered it just wasn't pushing out any power. Hit the power button again and boom, PDU back on and then switches. Oh boy is my manager going to have it out with the wiring companies boss as to why in the holy gently caress was he touching any equipment! The other weird thing, which we've had this issue before, is our network admin had no clue what was going on. I thought all our UPS's were managed and he certainly should have gotten some sort of alert that, hey, this UPS has 0 draw on it, something might be up! But no, apparently he got no such alert. Also, the PDU has dual power supplies in it, both were plugged into the same UPS. 
|
|
#
?
Apr 24, 2019 20:22
|
|
|
mattfl posted:Also, the PDU has dual power supplies in it, both were plugged into the same UPS.
|
|
#
?
Apr 24, 2019 21:10
|
|
|
I've seen it done that way where the setup was to have redundant racks rather than redundant power and network connections to each rack. Simplifies setup, costs a fuckton.
|
|
#
?
Apr 24, 2019 23:17
|
|
|
TIL that apparently if you have a wrongly encoded enough wav file playing as a menu prompt in Voicemail Pro, it will work fine a few times and then spontaneously double in file size and transform into a corrupted cacophony that sounds like an audio log from Hell.
|
|
#
?
Apr 25, 2019 00:32
|
|
|
Late last week I had a ticket cut directly to me from HR prompting me to complete a background check. This is strange because I’ve been here for a few years now and have never been asked to submit to one. Doing some digging and asking a few people I found out the reason: apparently when I copied my permissions from my mentor’s credentials back when there were only a few of us in my role globally, I was granted root access to one of the foundational services we offer. Meaning, not only did I have access to all kinds of internal data, I had the ability to turn the service completely off at a global level. I’ve hosed up by turning off the wrong server before, but having the ability to turn off or break a service for the entire world is next level paranoia-inducing. I could not get the real service admins to remove me from the role group fast enough.
|
|
#
?
Apr 25, 2019 04:37
|
|
|
Agrikk posted:Late last week I had a ticket cut directly to me from HR prompting me to complete a background check. This is strange because I’ve been here for a few years now and have never been asked to submit to one. And then I started to think about it, and I've got substantial access to several hundred million dollars in corporate assets. Had a low-grade anxiety attack. I try not to think about that stuff too much. In some ways, working at a small business is better; the stakes for your fuckups are so much lower.
|
|
#
?
Apr 25, 2019 04:56
|
|
|
Agrikk posted:I’ve hosed up by turning off the wrong server before, but having the ability to turn off or break a service for the entire world is next level paranoia-inducing.
|
|
#
?
Apr 25, 2019 05:20
|
|
|
Agrikk posted:Late last week I had a ticket cut directly to me from HR prompting me to complete a background check. This is strange because I’ve been here for a few years now and have never been asked to submit to one. phsh. I have root access to the servers that run the entire core business. If they're down, it's 7 figures/hour. NBD, really. 
|
|
#
?
Apr 25, 2019 05:25
|
|
|
Agrikk posted:Late last week I had a ticket cut directly to me from HR prompting me to complete a background check. This is strange because I’ve been here for a few years now and have never been asked to submit to one. https://aws.amazon.com/message/41926/
|
|
#
?
Apr 25, 2019 06:09
|
|
|
Entropic posted:TIL that apparently if you have a wrongly encoded enough wav file playing as a menu prompt in Voicemail Pro, it will work fine a few times and then spontaneously double in file size and transform into a corrupted cacophony that sounds like an audio log from Hell. This reminded me of extension 666. You definitely want the volume turned down for this one.
|
|
#
?
Apr 25, 2019 12:09
|
|
|
Thanatosian posted:I was discussing world travel and stuff with a friend, and he mentioned being kidnapped, and I started to say "no one would ever want to kidnap me..." I won't visit Russia, China, and some of the older eastern-bloc countries for similar reasons.
|
|
#
?
Apr 25, 2019 12:15
|
|
|
A vendor on a conference call is getting very upset that we're being mean to them. I think we're allowed to be mean to vendors after their critical service has been broken for 36 hours and they have yet to do anything to fix it.
|
|
#
?
Apr 25, 2019 12:50
|
|
|
Aunt Beth posted:I‘m always kind of awestruck at scale like this. What service? Or if you can’t give the name of the service what are some things we might notice go offline? Pretty sure he works for Amazon and deals with AWS, so..... I could be getting computer touchers mixed up though.
|
|
#
?
Apr 25, 2019 14:08
|
|
|
Alighieri posted:Pretty sure he works for Amazon and deals with AWS, so..... I could be getting computer touchers mixed up though. No, you're right. Agrikk has made no secret of it here.
|
|
#
?
Apr 25, 2019 14:16
|
|
|
Like this, yeah. But for the whole world at once. Sort of like at the end of Ready Player One when the kid wins the game and becomes all-powerful and the old guy shows him the button that turns off OASIS for good. I’m real glad I don’t have that access anymore.
|
|
#
?
Apr 25, 2019 14:52
|
|
|
I was studying up on BGP yesterday and was kind of disturbed to learn that a single incompetent administrator can take down the entire internet. Then I was disturbed to learn that it's already happened e: I didn't find any documentation on it but I do remember an outage we had a while back where seemingly random customers were unable to reach random websites. I found out during the RCA that verizon had accidentally advertised a ton of extra BGP routes and it filled up the routing table on a bunch of older Cisco routers. So a bunch of the world's WAN routers just forgot how to route a portion of the internet. Renegret fucked around with this message at 15:05 on Apr 25, 2019 |
|
#
?
Apr 25, 2019 15:00
|
|
|
Renegret posted:I was studying up on BGP yesterday and was kind of disturbed to learn that a single incompetent administrator can take down the entire internet. https://www.zdnet.com/article/some-internet-outages-predicted-for-the-coming-month-as-768k-day-approaches/ It might happen again soon.
|
|
#
?
Apr 25, 2019 15:11
|
|
|
Alighieri posted:https://www.zdnet.com/article/some-internet-outages-predicted-for-the-coming-month-as-768k-day-approaches/ it's worth mentioning that I didn't look for information about it that hard either  This article is both terrifying, and useful, because I have an interview tomorrow for the network team and I can probably make myself look good by bringing this up.
|
|
#
?
Apr 25, 2019 15:14
|
|
|
Renegret posted:I was studying up on BGP yesterday and was kind of disturbed to learn that a single incompetent administrator can take down the entire internet.
|
|
#
?
Apr 25, 2019 15:44
|
|
|
Renegret posted:I was studying up on BGP yesterday and was kind of disturbed to learn that a single incompetent administrator can take down the entire internet. ...at my old job they still told the tale of one broken fuse breaking the internet for 40% of Europe.
|
|
#
?
Apr 25, 2019 15:55
|
|
|
DONT TOUCH THE PC posted:...at my old job they still told the tale of one broken fuse breaking the internet for 40% of Europe. Don't forget that shutting down power plants in Eastern Europe made the clocks in England run slow.
|
|
#
?
Apr 25, 2019 16:16
|
|
|
Kurieg posted:Don't forget that shutting down power plants in Eastern Europe made the clocks in England run slow. Dafuq?
|
|
#
?
Apr 25, 2019 16:21
|
|
|
Agrikk posted:Dafuq? https://www.theverge.com/2018/3/8/17095440/europe-clocks-running-slow-electricity-frequency-kosovo-serbia
|
|
#
?
Apr 25, 2019 16:34
|
|
|
England was actually ok. They're connected to the mainland by HVDC and operate their internal grid frequency independently from the rest of Europe. The rest of the continent was affected though. The organization that oversees the grid put out a passive aggressive press release asking to keep political issues out of the power grid.
|
|
#
?
Apr 25, 2019 16:41
|
|
|
The worst I’ve seen was a voice router losing its IP connection (so no calls) because the Ethernet cable had creeped slightly out on its own. Literally touched the cable to push it back in and all was well. Good thing there was an HA router up and running.
|
|
#
?
Apr 25, 2019 16:54
|
|
|
Agrikk posted:I’ve hosed up by turning off the wrong server before, but having the ability to turn off or break a service for the entire world is next level paranoia-inducing. At least you've never inadvertently terminated your company's Japanese CEO? In my defense, he was on my list!...I realize, typing this, that it makes me sound like I'm an international assassin and not a computer toucher. Thanatosian posted:I was discussing world travel and stuff with a friend, and he mentioned being kidnapped, and I started to say "no one would ever want to kidnap me..." Yeeeah. My company has an international hotline that I describe to people as, "Basically, you call, and Liam Neeson shows up." When I went to Ukraine last year with my band, I had to contact our international security team and get their assessment of the area. It wasn't bad enough that I would have to leave my work assets at home, but I didn't take anything just in case. We were in Odessa for a few days, and that's shading into the separatist areas of the country these days. There are a few countries we're prohibited from taking work assets into, and we have temporary laptops and phones for those people. I don't THINK I'm a target for anything hinky, but I definitely don't wear my  swag in public much, either.
|
|
#
?
Apr 25, 2019 17:10
|
|
|
Dirt Road Junglist posted:At least you've never inadvertently terminated your company's Japanese CEO? Checks out.
|
|
#
?
Apr 25, 2019 17:46
|
|
|
SamsCola posted:England was actually ok. They're connected to the mainland by HVDC and operate their internal grid frequency independently from the rest of Europe. The rest of the continent was affected though. The organization that oversees the grid put out a passive aggressive press release asking to keep political issues out of the power grid. Ireland, Northern Ireland, Wales and Scotland were also fine! 
|
|
#
?
Apr 25, 2019 17:52
|
|
|
I imagine people who do not use kitchen appliances to manage their time were okay too.
|
|
#
?
Apr 25, 2019 18:07
|
|
|
feedmegin posted:Ireland, Northern Ireland, Wales and Scotland were also fine! Don't forget the Isle of Man
|
|
#
?
Apr 25, 2019 20:36
|
|
|
feedmegin posted:Ireland, Northern Ireland, Wales and Scotland were also fine! I went deep down the rabbit hole of research for a few days after hearing about that whole power time thing. I was living in Ireland at the time and discovered that the Irish power authority has a website with live details of the power generation by type of generator in the country (among other neat into) and details on the HVDC links to the UK. Real cool stuff. I guess it's related to IT in that I was goofing off at work?
|
|
#
?
Apr 25, 2019 21:57
|
|
|
This is a neat app http://www.cs.ox.ac.uk/people/alex.rogers/gridcarbon/ National Grid also publish data https://extranet.nationalgrid.com/RealTime and it's made more accessible here http://gridwatch.co.uk/
|
|
#
?
Apr 25, 2019 22:56
|
|
|
Sheep posted:This reminded me of extension 666. You definitely want the volume turned down for this one. The glitch I had was much more literally hellish sounding. This was the first thing callers got when they called the main number. At very high volume.
|
|
#
?
Apr 26, 2019 01:04
|
|
|
Dirt Road Junglist posted:At least you've never inadvertently terminated your company's Japanese CEO? Gotta be honest, I'd dawdle as long as possible on fixing that.
|
|
#
?
Apr 26, 2019 01:18
|
|
|
A Russian email address sent our service desk address a little test email at 4:48 PM yesterday. After the service system sent back a positive automated response, the flood gates opened. Between 5:04 PM and 6:06 AM, we received over 1000 false help tickets advertising everything from pharmaceuticals to mail order brides.
|
|
#
?
Apr 26, 2019 01:31
|
|
|
Did you try turning her off and on again?
|
|
#
?
Apr 26, 2019 01:33
|
|
|
|
| # ? Apr 19, 2024 08:25 |
|
|
my cat is norris posted:A Russian email address sent our service desk address a little test email at 4:48 PM yesterday. The mistake was in accepting mail from Russia, hth?
|
|
#
?
Apr 26, 2019 01:36
|
|
