Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us $3,400 per month for bandwidth bills alone, and since we don't believe in shoving popup ads to our registered users, we try to make the money back through forum registrations.
«1275 »
  • Post
  • Reply
H110Hawk
Dec 28, 2006
Can't install Windows?
BUY APPLE


Docjowles posted:

At a past job, we had some bastion hosts you had to jump through to reach production. The NOC team sat on those jump boxes all day, and would loving CONSTANTLY space out and reboot them instead of the actual host they meant to work on. We eventually literally replaced every command we could think of (suhtdown, reboot, halt, etc) with a symlink to a script that said "HEY NUMBNUTS THIS IS THE JUMP BOX. Did you really mean to reboot it? y/n" and then called the real command.

Kind of a silly hack but dammit, it cut down on the unintended reboots!

Mollyguards are a time honored tradition. The better ones make them repeat a challenge word to reboot. If you have a "remote reboot command" you script up you can even install the mollyguard everywhere. That has the added benefit of you can write into a database datetime,username,server,comment from the reboot command.

Adbot
ADBOT LOVES YOU

Docjowles
Apr 9, 2009



Nice, hadn't heard that term for it. Apparently there's even a package of that name designed to guard against accidental reboots, lol

http://manpages.ubuntu.com/manpages...ly-guard.8.html

H110Hawk
Dec 28, 2006
Can't install Windows?
BUY APPLE


Docjowles posted:

Nice, hadn't heard that term for it. Apparently there's even a package of that name designed to guard against accidental reboots, lol

http://manpages.ubuntu.com/manpages...ly-guard.8.html

Kids these days!

Zorak of Michigan
Jun 10, 2006

Waiting for his chance

On one hand there's really no cogent argument against something like a mollyguard. If it reduces the risk of an accident that would impact availability, it's good. On the other hand, if an employee is repeatedly careless while running commands with root privileges, mollyguard isn't the whole solution. It might keep him from rebooting the wrong server, but who knows what else the guy is doing carelessly? There's a basic level of care and focus you need to apply when doing things as root, and if you regularly fall short of that level, operations is not a good place for you.

Thanks Ants
May 21, 2004

Bless You Ants, Blants



Fun Shoe

You shouldn't avoid putting easy methods (with little to no downsides) in place to stop people making mistakes just because they might become a crutch one day. If people are generally careless then address that separately through your change control processes, training, discipline policies, improved ways to roll back changes etc.

GnarlyCharlie4u
Sep 23, 2007

I have an unhealthy obsession with motorcycles.

Proof


https://www.privacyfly.com/articles/ncix_breach/

This is absolutely amazing.

quote:

Millions of Canadian and American consumers are now at risk thanks to a series of shady backroom deals that have resulted in records detailing 15 years of business being sold.

August 1, 2018. A rare sunny day in rain ridden Vancouver, British Columbia. Typical of my introverted lifestyle, I found myself indulging my passion for used computer hardware by scouring Craigslist. Post after post of monotonous listings began to blend together as an intriguing title caught my eye. “NCIX Database Servers - $1500 (Richmond BC)”.

He proceeded to tell me that not only did he have the network drive that I was inquiring about, but he also possessed NCIX’s entire server farm from the east coast which was shipped back to their Richmond warehouse several months ago.

NCIX had abandoned the hardware when they failed to pay a past due rent total of $150,000.

September 5th, 2018. I arrived at the Warehouse midday this time with slightly more insight into the software required to open and analyze the various files strewn among the 109 hard drives. I once again was ushered upstairs, where Jeff had prepared two supermicro server’s running StarWind iSCSI Software and one of the 300 desktops as a sample. I first sat down at the desktop and discovered that it was used by a former NCIX employee named Chadwick Ma. The computer contained a treasure trove of confidential data including credentials, invoices, photographs of customers ID’s, Bills, and Mr. Ma’s T4 among other files.

The nciwww database contained a thousand records from affiliates listing plain text passwords, addresses, names, and some financial data.
In another table of information, I found customer service inquiries including messages and contact information. There were also three hundred eighty-five thousand names, serial numbers with dates of purchase, addresses, company names, email addresses, phone numbers, IP addresses and unsalted MD5 hashed passwords. The database also contained full credit card payment details in plain text for two hundred and fifty-eight thousand users between various tables.

Jonny Nox
Apr 25, 2008


I love the name Mollyguard


wiktionary posted:

Originally a Plexiglas cover improvised for the Big Red Switch on an IBM 4341 mainframe after a programmer's toddler daughter (named Molly) tripped it twice in one day. Later generalised to covers over stop/reset switches on disk drives and networking equipment.

Thanks Ants
May 21, 2004

Bless You Ants, Blants



Fun Shoe

"Hey we're making this change in two weeks, but the replacement is in production now! Here's how to move over to it, give us a shout if you get stuck"
"Just a reminder that this legacy system is going away in two weeks, the new way to do exactly the same thing is explained here, our support team can take you through it if you have any issues"
"This thing is happening in three days"
"Tomorrow the old system is going away"
"Ok, that's all done now. If you're still set up to use the old service then get in touch and somebody can take you through the change, alternatively see this documentation"

Email CCing their boss, their bosses boss
"I've not been able to get into this for four days now so haven't been able to get any work done, what's going on?"

Every. loving. Time.

GreenNight
Feb 19, 2006
Turning the light on the darkest places, you and I know we got to face this now. We got to face this now.

That's when I detail each email they received concerning this with dates and times via a CC to everyone they CC'd. I then tell them I confirmed that they received each email and ask them which part of the process is confusing so that I can make it better in the future.

Thanks Ants
May 21, 2004

Bless You Ants, Blants



Fun Shoe

This isn't a CYA exercise this time, the people that this person is complaining to also received the notifications and were involved in assisting people make the change or at least get help if needed.

MC Fruit Stripe
Nov 26, 2002

around and around we go


No GreenLight's just saying be helpful in a condescending way, which is an approach I enjoy myself.

Thanks Ants
May 21, 2004

Bless You Ants, Blants



Fun Shoe

Ah yeah the "let me know which of the steps you're stuck on and we'll figure it out" response does get used a fair bit.

GreenNight
Feb 19, 2006
Turning the light on the darkest places, you and I know we got to face this now. We got to face this now.

I just have a hard time putting up with people who can't read emails then try to throw IT under the bus. But everyone is busy and reading is hard.

18 Character Limit
Apr 6, 2007

Screw you, Abed;
I can fix this!


Nap Ghost

GreenNight posted:

I just have a hard time putting up with people who can't read emails then try to throw IT under the bus. But everyone is busy and reading is hard.

Throwing IT under the bus is such a time-honored tradition that many groups have it as step 1 of troubleshooting.

Adbot
ADBOT LOVES YOU

GreenNight
Feb 19, 2006
Turning the light on the darkest places, you and I know we got to face this now. We got to face this now.

I just got back from a 10 day vacation. Not a shitshow thankfully but yay for a headache.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply
«1275 »