Working in IT 3.0: xp_cmdshell

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Working in IT 3.0: xp_cmdshell

MC Fruit Stripe: Nov 26, 2002; around and around we go

Since this came up within the context of my talking about a mistake - I was referring to an upgrade of dozens of switches with a single port misconfigured. There is no amount of planning, ever, which prevents that. That doesn't mean you should be like "well since something's going to break let's just throw it in", but the idea that a bit more planning would have meant no mistakes at all is ludicrous.

# ? Nov 16, 2016 12:47

Adbot: ADBOT LOVES YOU

# ? Apr 16, 2024 21:10

3 Action Economist: May 22, 2002; Educate. Agitate. Liberate.

If part of your proper plan is to follow a checklist, and within that checklist is to verify the configuration, yes, it could have been prevented.

# ? Nov 16, 2016 12:52

milk milk lemonade: Jul 29, 2016

devmd01 posted:

I woke up at 4 this morning and worked until 530, got 3 servers migrated. Down to 42, and I'm building out a new infrastructure for one environment, so we won't have to migrate 10 vms.

I'm working constantly, whether it's Sunday morning migrations or just vmotioning servers after the kid is in bed to keep replication churning 24/7. I'm not afraid to put in the work, but this can only be a short term thing, especially with twins on the way.

Do you have any kids yet? I can tell you that sort of schedule is grounds for a divorce when you have childcare responsibilities

# ? Nov 16, 2016 13:37

MC Fruit Stripe: Nov 26, 2002; around and around we go

Colonial Air Force posted:

If part of your proper plan is to follow a checklist, and within that checklist is to verify the configuration, yes, it could have been prevented.

As soon as my eyes roll back around to the front of my head I'll have a more substantive reply for you.

# ? Nov 16, 2016 13:41

Collateral Damage: Jun 13, 2009

We have a core firewall replacement coming up. I'm not directly involved in the job, but I've been shanghaied into post-change testing.

This is the network security team's change plan:

1. Unplug and power off the old firewall cluster. (15 minutes)
2. Unbox and rack the new firewall cluster. (30 minutes)
3. Connect management interface.
4. Apply basic configuration. (30 minutes)
5. Build ruleset identical to old firewall (4 hours)
6. Connect main interfaces.
7. Test ruleset and flows. (2 hours)
8. Unrack old firewalls. (15 minutes)

:psyboom:

Did I mention this is the core EMEA firewall for a global company with tens of thousands of employees?

Collateral Damage fucked around with this message at 14:00 on Nov 16, 2016

# ? Nov 16, 2016 13:57

Sefal: Nov 8, 2011; Fun Shoe

Told the place I'm detached at that i'm leaving for another job. he was really cool about it.
While he was sad that i'm leaving. He completely supported my decision and said that that is probably the best thing I could have done. "And hey maybe next year you'll come back here. The doors of <this company> will always be open for you."
He's also going to keep quiet about this until I've told my boss.

Now I just gotta sign the papers and tell my boss.

# ? Nov 16, 2016 14:02

rafikki: Mar 8, 2008; I see what you did there. (It's pretty easy, since ducks have a field of vision spanning 340 degrees.)

~SMcD

Collateral Damage posted:

We have a core firewall replacement coming up. I'm not directly involved in the job, but I've been shanghaied into post-change testing.

This is the network security team's change plan:

1. Unplug and power off the old firewall cluster. (15 minutes)
2. Unbox and rack the new firewall cluster. (30 minutes)
3. Connect management interface.
4. Apply basic configuration. (30 minutes)
5. Build ruleset identical to old firewall (4 hours)
6. Connect main interfaces.
7. Test ruleset and flows. (2 hours)
8. Unrack old firewalls. (15 minutes)

Did I mention this is the core EMEA firewall for a global company with tens of thousands of employees?

I like how step 5 occurs after taking out the old firewalls.

# ? Nov 16, 2016 14:07

MC Fruit Stripe: Nov 26, 2002; around and around we go

Why would you not just have the config in place on the new cluster? And I mean, how could anyone not get there without having to be told that?

It's scenarios like that which make me go into an interview thinking "please, please, please ask me about migrating infrastructure, I will apparently blow your mind with my common sense"

MC Fruit Stripe fucked around with this message at 14:28 on Nov 16, 2016

# ? Nov 16, 2016 14:21

Siochain: May 24, 2005; "can they get rid of any humans who are fans of shitheads like Kanye West, 50 Cent, or any other piece of crap "artist" who thinks they're all that?

And also get rid of anyone who has posted retarded shit on the internet."

Colonial Air Force posted:

If part of your proper plan is to follow a checklist, and within that checklist is to verify the configuration, yes, it could have been prevented.

And, once again, human error can strike.

You literally cannot prevent it. You can minimize it and reduce it, but you can never fully remove it.

# ? Nov 16, 2016 14:26

Internet Explorer: Jun 1, 2005

DESTROY ALL HUMANS

# ? Nov 16, 2016 14:54

pixaal: Jan 8, 2004; All ice cream is now for all beings, no matter how many legs.

Siochain posted:

And, once again, human error can strike.

You literally cannot prevent it. You can minimize it and reduce it, but you can never fully remove it.

That is what the PM wants, a reduced chance of it happening again. Peer review, doing changes during the work day so someone senior is there to step in, doing it during the work day so someone isn't up at 2AM fixing the problem, likely sleep deprived and a few drinks in.

They might every well reject the prevention steps, at that point yes you are already at the acceptable risk point.

# ? Nov 16, 2016 14:55

Super Slash: Feb 20, 2006; You rang ?

Siochain posted:

You literally cannot prevent it. You can minimize it and reduce it, but you can never fully remove it.

Exactly this, the absolute best you can do is make a process which mitigates failures.

A checkbox isn't going to stop someone tripping over and ripping a cable out a slot, or typing a 3 instead of a 2, etc etc.

# ? Nov 16, 2016 14:56

Thanks Ants: May 21, 2004; #essereFerrari

Farking Bastage posted:

Oh how I love new hardware day. HP has discontinued the venerable 3500-YL-48GPOE, so these are the replacement. Aruba 3810's, but with 40GBE stacking modules instead of 10.

I hope that piece of poo poo in the background is going in a dumpster

# ? Nov 16, 2016 15:07

mayodreams: Jul 4, 2003; Hello darkness,
my old friend

Collateral Damage posted:

We have a core firewall replacement coming up. I'm not directly involved in the job, but I've been shanghaied into post-change testing.

This is the network security team's change plan:

1. Unplug and power off the old firewall cluster. (15 minutes)
2. Unbox and rack the new firewall cluster. (30 minutes)
3. Connect management interface.
4. Apply basic configuration. (30 minutes)
5. Build ruleset identical to old firewall (4 hours)
6. Connect main interfaces.
7. Test ruleset and flows. (2 hours)
8. Unrack old firewalls. (15 minutes)

Did I mention this is the core EMEA firewall for a global company with tens of thousands of employees?

I don't even. This is a case study in doing it wrong.

# ? Nov 16, 2016 15:19

Farking Bastage: Sep 22, 2007; Who dey think gonna beat dem Bengos!

Thanks Ants posted:

I hope that piece of poo poo in the background is going in a dumpster

Oh I'm going to office space that loving thing.

In other news, client called in with "network" problems. Our clients have enough autonomy that they always end up with some dummy switch on the network that wrecks poo poo, but we can't tell them not to, just bill the crap out of them when they inevitably gently caress up.

STP Enabled : Yes
Force Version : MSTP-operation
IST Mapped VLANs : 1-4094
Switch MAC Address : 001b3f-dca280
Switch Priority : 32768
Max Age : 20
Max Hops : 20
Forward Delay : 15

Topology Change Count : 15,529
Time Since Last Change : 16 hours

:suicide:

e: that's not good...

code:

3500yl-48g-02# show mod

 Status and Counters - Module Information

  Chassis: 3500yl-48G J8693A         Serial Number:   
                              Allow V1 Modules:   Yes


                                                                       Core  Mod
  Slot  Module Description                     Serial Number  Status   Dump  Ver
  ----- -------------------------------------- -------------- -------- ----- ---
  A                                                           Failed   -


3500yl-48g-02#

Farking Bastage fucked around with this message at 16:05 on Nov 16, 2016

# ? Nov 16, 2016 15:32

nominal: Oct 13, 2007; I've never tried dried apples.
What are they?; Pork Pro

Saw my boss was installing a fourth monitor today.

Me: "What's the matter, not buried in enough data yet?"
Him: "Nope, this isn't even for me. This is for the GRAND ILLUSION."
Me: ????

Turns out, way up the chain so far that they are almost guaranteed to have no idea what's going on, someone decided to tell one of our contracts about this amazing piece of software the call center uses to track events in real time by geographic location so we can instantly dispatch a solution, probably using *the cloud* or something, I don't know.

I'm in the call center.

We use no such software. There are no plans for there to BE any such software.

Apparently, they've decided to wrangle up some sort of thing that APPEARS to function as that sort of software, and then give him a dedicated monitor for it right outside the call center door so he can appear to be on top of the situation during a tour tomorrow. The second they leave, the monitor gets taken down and WE NEVER SPEAK OF THIS AGAIN.

I am somehow simultaneously disgusted, terrified, and strangely impressed by this

# ? Nov 16, 2016 18:14

CLAM DOWN: Feb 13, 2007

nominal posted:

Saw my boss was installing a fourth monitor today.

Me: "What's the matter, not buried in enough data yet?"
Him: "Nope, this isn't even for me. This is for the GRAND ILLUSION."
Me: ????

Turns out, way up the chain so far that they are almost guaranteed to have no idea what's going on, someone decided to tell one of our contracts about this amazing piece of software the call center uses to track events in real time by geographic location so we can instantly dispatch a solution, probably using *the cloud* or something, I don't know.

I'm in the call center.

We use no such software. There are no plans for there to BE any such software.

Apparently, they've decided to wrangle up some sort of thing that APPEARS to function as that sort of software, and then give him a dedicated monitor for it right outside the call center door so he can appear to be on top of the situation during a tour tomorrow. The second they leave, the monitor gets taken down and WE NEVER SPEAK OF THIS AGAIN.

I am somehow simultaneously disgusted, terrified, and strangely impressed by this

Lol this is pretty awesome

# ? Nov 16, 2016 18:18

Thanks Ants: May 21, 2004; #essereFerrari

Do it as a recorded tour on Google Earth for extra 'looks like it does on TV' appeal.

# ? Nov 16, 2016 18:20

MC Fruit Stripe: Nov 26, 2002; around and around we go

That is friggin brilliant, and yes, seconding the Google Earth request. You can do 'tours' on Google Earth and set it to just zoom in on random parts of the world. Just have it automatically zipping around the planet like it's scanning for outages.

# ? Nov 16, 2016 18:46

Sefal: Nov 8, 2011; Fun Shoe

nominal posted:

Saw my boss was installing a fourth monitor today.

Me: "What's the matter, not buried in enough data yet?"
Him: "Nope, this isn't even for me. This is for the GRAND ILLUSION."
Me: ????

Turns out, way up the chain so far that they are almost guaranteed to have no idea what's going on, someone decided to tell one of our contracts about this amazing piece of software the call center uses to track events in real time by geographic location so we can instantly dispatch a solution, probably using *the cloud* or something, I don't know.

I'm in the call center.

We use no such software. There are no plans for there to BE any such software.

Apparently, they've decided to wrangle up some sort of thing that APPEARS to function as that sort of software, and then give him a dedicated monitor for it right outside the call center door so he can appear to be on top of the situation during a tour tomorrow. The second they leave, the monitor gets taken down and WE NEVER SPEAK OF THIS AGAIN.

I am somehow simultaneously disgusted, terrified, and strangely impressed by this

This is amazing

# ? Nov 16, 2016 18:49

Agrikk: Oct 17, 2003; Take care with that! We have not fully ascertained its function, and the ticking is accelerating.

Farking Bastage posted:

Oh how I love new hardware day. HP has discontinued the venerable 3500-YL-48GPOE, so these are the replacement. Aruba 3810's, but with 40GBE stacking modules instead of 10.

The days that hardware arrives are literally the best work days ever. These are the days that you can point back to and say, "See? I built that. Before this, this didn't exist and now it does."

# ? Nov 16, 2016 19:06

MF_James: May 8, 2008; I CANNOT HANDLE BEING CALLED OUT ON MY DUMBASS OPINIONS ABOUT ANTI-VIRUS AND SECURITY. I REALLY LIKE TO THINK THAT I KNOW THINGS HERE

INSTEAD I AM GOING TO WHINE ABOUT IT IN OTHER THREADS SO MY OPINION CAN FEEL VALIDATED IN AN ECHO CHAMBER I LIKE

Agrikk posted:

The days that hardware arrives are literally the best work days ever. These are the days that you can point back to and say, "See? I built that. Before this, this didn't exist and now it does."

The first thing we do with new equipment is open up the case and marvel at the inner workings (provided it's NEW equipment we haven't already seen)

# ? Nov 16, 2016 19:08

Docjowles: Apr 9, 2009

MF_James posted:

The first thing we do with new equipment is open up [a support] case and marvel at the [DOA rate]

fixed for my experience

# ? Nov 16, 2016 20:15

Modulo16: Feb 12, 2014; "Authorities say the phony Pope can be recognized by his high-top sneakers and incredibly foul mouth."

nominal posted:

Saw my boss was installing a fourth monitor today.

Me: "What's the matter, not buried in enough data yet?"
Him: "Nope, this isn't even for me. This is for the GRAND ILLUSION."
Me: ????

Turns out, way up the chain so far that they are almost guaranteed to have no idea what's going on, someone decided to tell one of our contracts about this amazing piece of software the call center uses to track events in real time by geographic location so we can instantly dispatch a solution, probably using *the cloud* or something, I don't know.

I'm in the call center.

We use no such software. There are no plans for there to BE any such software.

Apparently, they've decided to wrangle up some sort of thing that APPEARS to function as that sort of software, and then give him a dedicated monitor for it right outside the call center door so he can appear to be on top of the situation during a tour tomorrow. The second they leave, the monitor gets taken down and WE NEVER SPEAK OF THIS AGAIN.

I am somehow simultaneously disgusted, terrified, and strangely impressed by this

This is awesome. Thirding the Google Earth thing.

# ? Nov 16, 2016 21:37

pixaal: Jan 8, 2004; All ice cream is now for all beings, no matter how many legs.

Why are we supporting ripping off customers? I get that a show needs to be made because sales is stupid and now you need to put on a show or lose your job for calling sales out.

# ? Nov 16, 2016 22:04

Judge Schnoopy: Nov 2, 2005; dont even TRY it, pal

"Why didn't you know about this outage? We confirmed you have that outage tracking system in place! Were you not paying attention to it, which is your job?!?"

# ? Nov 16, 2016 22:09

MF_James: May 8, 2008; I CANNOT HANDLE BEING CALLED OUT ON MY DUMBASS OPINIONS ABOUT ANTI-VIRUS AND SECURITY. I REALLY LIKE TO THINK THAT I KNOW THINGS HERE

INSTEAD I AM GOING TO WHINE ABOUT IT IN OTHER THREADS SO MY OPINION CAN FEEL VALIDATED IN AN ECHO CHAMBER I LIKE

Judge Schnoopy posted:

"Why didn't you know about this outage? We confirmed you have that outage tracking system in place! Were you not paying attention to it, which is your job?!?"

This is exactly what I envisioned happening, no calls coming in but everyone is down.

# ? Nov 16, 2016 22:22

devmd01: Mar 7, 2006; Elektronik
Supersonik

milk milk lemonade posted:

Do you have any kids yet? I can tell you that sort of schedule is grounds for a divorce when you have childcare responsibilities

Yep, a two year old. Wife and I are good, that's all I'll say.

# ? Nov 16, 2016 22:24

The Fool: Oct 16, 2003

My Softchoice rep just name-dropped Isreal's Ministry of Defense.

Not sure how I feel about that.

edit: context. The full sentence was "I am still working with <list of a half-dozen giant orginizations>, but I'll give you 100% of my focus!"

The Fool fucked around with this message at 22:54 on Nov 16, 2016

# ? Nov 16, 2016 22:51

milk milk lemonade: Jul 29, 2016

devmd01 posted:

Yep, a two year old. Wife and I are good, that's all I'll say.

Then you're already a pro haha

But man do those schedules burn when you have kids at home

# ? Nov 16, 2016 23:13

Walked: Apr 14, 2003

Offered a position with a 20k pay cut, but better insurance and 100% remote work. Still pays very well, and that time being with my kids instead of traffic is so valuable to me.

Plus I just got done finishing my basement with a dedicated sweet rear end office.

:getin:

# ? Nov 17, 2016 00:00

Proteus Jones: Feb 28, 2013

Colonial Air Force posted:

If part of your proper plan is to follow a checklist, and within that checklist is to verify the configuration, yes, it could have been prevented.

MC Fruit Stripe posted:

dozens of switches

See you in a couple days! Sorry can't put it into production yet.

God forbid you push to hundreds of switches.

# ? Nov 17, 2016 00:02

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Walked posted:

Offered a position with a 20k pay cut, but better insurance and 100% remote work. Still pays very well, and that time being with my kids instead of traffic is so valuable to me.

Plus I just got done finishing my basement with a dedicated sweet rear end office.

When I took my previous job, I took a 15k pay cut but still ended up ahead financially because of how much cheaper my insurance was. It's crazy how much the variation is.

# ? Nov 17, 2016 03:14

3 Action Economist: May 22, 2002; Educate. Agitate. Liberate.

flosofl posted:

See you in a couple days! Sorry can't put it into production yet.

God forbid you push to hundreds of switches.

Then your plan should have started earlier. Timing is part of a project plan, too.

# ? Nov 17, 2016 03:18

Judge Schnoopy: Nov 2, 2005; dont even TRY it, pal

Vulture Culture posted:

When I took my previous job, I took a 15k pay cut but still ended up ahead financially because of how much cheaper my insurance was. It's crazy how much the variation is.

Especially if you work for government. Family of 5 through the US marketplace runs $1000 a month for deductibles that won't leave you afraid of going to the doctor. Drop that down to $250 a month for a loving $0 deductible HMO? Hell yeah I'll take a $10k paycut for that.

# ? Nov 17, 2016 03:20

Walked: Apr 14, 2003

Judge Schnoopy posted:

Especially if you work for government. Family of 5 through the US marketplace runs $1000 a month for deductibles that won't leave you afraid of going to the doctor. Drop that down to $250 a month for a loving $0 deductible HMO? Hell yeah I'll take a $10k paycut for that.

Yeah; what I'm looking at is not quite that drastic, but still noteworthy.

The biggest benefits, though, come from not having to slog through DC area traffic on a daily basis or sit in conference room meetings that are driven by buraucracy and in the end are meaningless wastes of time.

Not bitter; just fuckin stoked to spend more time with my kids instead of traffic and/or conference rooms :woop:

# ? Nov 17, 2016 03:37

The Fool: Oct 16, 2003

Judge Schnoopy posted:

Especially if you work for government. Family of 5 through the US marketplace runs $1000 a month for deductibles that won't leave you afraid of going to the doctor. Drop that down to $250 a month for a loving $0 deductible HMO? Hell yeah I'll take a $10k paycut for that.

Yeah. I just went to FEHB from private insurance. From $1200 a month for a family of three, to $150 a month and a $30 copay

# ? Nov 17, 2016 04:57

CLAM DOWN: Feb 13, 2007

Judge Schnoopy posted:

Especially if you work for government. Family of 5 through the US marketplace runs $1000 a month for deductibles that won't leave you afraid of going to the doctor. Drop that down to $250 a month for a loving $0 deductible HMO? Hell yeah I'll take a $10k paycut for that.

*twitches canadianly*

# ? Nov 17, 2016 05:00

KennyTheFish: Jan 13, 2004

flosofl posted:

See you in a couple days! Sorry can't put it into production yet.

God forbid you push to hundreds of switches.

I may be off track a bit here, but with hundreds would you not be building the configurations algorithmically from some sort of configuration engine?

# ? Nov 17, 2016 05:16

Adbot: ADBOT LOVES YOU

# ? Apr 16, 2024 21:10

Dreyvas: Jan 13, 2014

Walked posted:

Yeah; what I'm looking at is not quite that drastic, but still noteworthy.

The biggest benefits, though, come from not having to slog through DC area traffic on a daily basis or sit in conference room meetings that are driven by buraucracy and in the end are meaningless wastes of time.

Not bitter; just fuckin stoked to spend more time with my kids instead of traffic and/or conference rooms

Welcome to the glorious WFH master race. You'll never want to go back.

# ? Nov 17, 2016 06:16

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Working in IT 3.0: xp_cmdshell

«‹›2887 »