Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
Some people were discussing ease of migrating to certain areas a little while back.

I'll chime in and say that right now moving to New Zealand is pretty easy for anyone with a degree and a couple years work experience in any part of the IT industry as long as you're of good health. You don't even need a current job offer, it just helps.

You can find out how over-qualified you are by visiting here: http://www.immigration.govt.nz/migrant/stream/work/skilledmigrant/default.htm and then check out the points calculator.

Warning: The site is really bad and is an example of how badly you're needed here.

Adbot
ADBOT LOVES YOU

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
Reading the bit on the last page about deployments was funny. I too have been running most of the major deployments here for the last 2 years and a bit, and we're taking our tentative steps into the developers doing small deploys themselves. Long road to go down before we have a system that Docjowles describes, but then I think we were doing similar stuff back 7 years ago when we were the size of his company, and our CTO was also the head developer and main Ops guy (and his desktop was the deploy machine).


My manager also has me reading through The Phoenix Project right now, what's everyone's opinion on it? Personally I would have quit by chapter 4.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

AutoArgus posted:

Had a client once that had a rule: Only vice presidents and above are allowed to have even potential access to customer data.

This was a gigantic multistate bank with hundreds of workers in its IT division. One building was basically a shopping mall full of VPs of telephony and network storage and poo poo. Vice presidents of helpdesk. :psyduck:

Wait, don't bank tellers have access to customer data? Was the person taking your cash over the counter a Vice President of Customer Service?

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

Che Delilas posted:

You're probably going to run into a lot of stupid procedures or ways of doing things that are obviously inefficient or costing the company money or something that you have the perfect solution for. Ignore the impulse to try and fix these things, and just do the job you were brought in to do. Nobody likes the new guy coming in and telling them how wrong they're doing everything, even if it's the truth (especially if it's the truth).

I'm going to disagree with this. It depends a lot on the culture you're working in, I just hired 3 new guys and I'm letting them run wild with crazy solutions to problems that I've either not had time to deal with, or I haven't been able to see them as problems due to being here too long etc. Some of what they do may end up getting shot down, but for now I want to harness their newbie enthusiasm as much as possible.

In saying that, don't just say "This is dumb, do it this way instead". You need to ask WHY they are doing it that way? It could be a bad reason, such as "We did this due to a software bug for an application we no longer use, and then it was just habit", in which case your suggested improvement should be welcome. Or they could be doing a dumb thing for a very valid reason, such as "legal reasons" which in medical IT you'll probably come across a lot.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
Our HR department recently rebranded as the "Human department" and I've yet to decide if that's better or not. I was definitely in the "don't call me a resource" camp.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

the spyder posted:

Yea, it's a complete mess. All 100% custom built "gamer" desktops ranging from 3 months - 5 years old. No WDS, WSUS, software management. Ugg. At least it's not my problem. I'm just not seeing this as a good idea.

It works fine as long as the FNG they hired is a mature, responsible adult. In my first real admin job I did just fine with being tasked with "Find us an appropriate desktop that we'll buy 6 of for any new people", then I submitted it to my boss to make the purchase order.

Seeing a GAMER XXXTREME ALIENWARE Laptop in the request should have been that manager's first clue that he needed to fire that guy immediately while the probation period was still active.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

SSH IT ZOMBIE posted:

Probably that chain steakhouse. Wining and dining on your vendors' dime is totally a part of working in IT. :burger:
Edit: Interesting idea...

A certain part of IT is how well you're connected to the community in your area of expertise. Ie, going to user groups, speaking at them, giving demonstrations, reference calls with other customers, etc. If you're known, you can command a higher salary, and have connections if you're looking for new work.

:spergin: It's not for everyone, though.

There are varying levels of it. I never went to a user group, but I would never have gotten the awesome job I have now if not for the fact that I happened to accept a flatmate applicant who happened to work for this start up I'd never heard of.

Heck just participating in these threads has gotten goons employed.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

JHVH-1 posted:

Anyone else using Pagerduty and have some tips/tricks/l33t h4x?

We use it, works pretty well. Did you have any specific questions? I never set it up, but I am a user of it and have played around with it's scheduling tools

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

JHVH-1 posted:

It seems like the place I am at has all that set up in a way that works. They have groups for the application admins and certain special apps that go to the right people. The stuff I am going to be on call for comes from nagios and when its acknowledged there it automatically acknowledges it in Pagerduty. Seems like the way they use it is sms or push notification in like 5/10 minutes and then if thats not answered then it calls them to wake people up, and then if nothing from that it goes to a secondary on call.
This place is weird, and they have some big name clients but things are at least spread out or designed for failover so anytime a single thing has a problem its usually not a showstopper.

How did you get Nagios acks to forward into pager duty? I thought that wasn't possible? Or am I thinking about pager duty not forwarding acks to Nagios?

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
I always end my interviews by saying "As you're on the market, you probably have other applications going as well. If any of them start to heat up, please keep us in the loop and we'll do what we can to try and keep our hiring process in sync with you, so that you can make a clear decision at the end"

Tell the jobs you want that you've got a strong looking candidate already, but you're very interested in what they may have to offer. If it ends up that you were only in their 'maybe' pile, they'll probably give you a quicker No answer and you'll be able to move on. Buying yourself a little bit of extra time with the 'safety' job by telling them you're monstrously sick, incapable of talking, and probably contagious, wouldn't hurt matters though as far as I can tell.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

Fiendish Dr. Wu posted:

What'd you do with Splunk? I only recently started playing with it to grab twitter feeds and make pretty charts and graphs, but I know it can be used for so much more.

I look after a bunch of web servers. It's invaluable for finding out things during an outage like "Ok, I can clearly see that on these servers, the amount of log entries they were generating suddenly changed at 10:30pm" "Let's drill down closer and see if any particular error message started occurring then" "Oh hey, this cryptic message only started showing up at that time and had been completely unknown beforehand, it must be related!"

Or the classic "Customers are reporting that the page fails to load, but only sometimes" "Ok, a quick search across the web servers, is there any particular one in my load balanced pool that's throwing errors" "Number #28 is the culprit! Pull it out of the pool and figure out why it broke afterwards!"

Splunk is how we discovered the root cause for an issue I was dealing with until 4:30am this morning. Really glad we're flexible with hours and I was able to sleep in till midday. Unfortunately for my manager, he had a taxi picking him up to go to the airport at 5am. gently caress knows how his day went after that.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
Not all workplaces are poo poo guys, some of us have good jobs!

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
I once asked a co-worker to write out some regex for me while he was phone posting.


It didn't work :(

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

Tab8715 posted:

Are there any basic guidelines I should know when answering open-ended questions?

I was asked "A client is upset, how do you handle it?" or "An important server crashed and a client is losing thousands of dollars per minute, what do you do?".

I responded with more or less "I would let the customer speak, listen and not even interrupt if I knew the answer or even if it was their fault." and "I would safely and quickly stop whatever I'm doing, message my immediate supervisor, so they know what's going on, what I'm doing, where I am and start/join an outage bridge".

I was never hired for that gig but I did get up to the 3rd round of interviews :shrug:

Every place is going to have it's own ways of dealing with Big Emergencies, and how you think is best to handle them is going to be a result of where you've worked before hand.

I know I wouldn't like that answer, my preferred answer would be "fix it in whatever method is as fast as possible without having side effects that could be potentially worse than the current problem." If you're sitting there listening to someone yell, then the 5-10 minutes they're yelling is 5-10 minutes added to the outage. But I think that way because I work in an Ops team for a cloud application, something going wrong is going to be affecting thousands of people at once, and I don't deal with customers so I can be very short with people in getting the information I need. Someone who's from a background of a VAR or similar would probably prefer your answer as it's more about making the customer feel loved, while I care about the number of seconds the website is down for.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

Tab8715 posted:

Interesting, but which part of that are you disagreeing with? I was on a Product App. Support team which is more or less basically Operations. If an outage occurred, no matter how severe, we had to make a ticket, page out to development and management before we touched anything.

And how long did that take to do?

A couple months back I did something that took the site down. I went from doing that, having a gut feeling it was a bad idea, confirming my suspicions, figuring out the quickest way to fix, and implementing it in 5 minutes 45 seconds, and a big chunk of that time was me reuploading code via our deployment system.

During that time I fired off one email to the rest of the Ops team that was pretty much this verbatim:

Subject: I have hosed #site#
Content: fixing now, 5 minutes.

Getting everyone in to talk about the situation can wait until after the problem is fixed, and staff should be trusted enough to make the best decisions about the fix without having to ask everyone else first.

The most we do when it's not going to be quick is let the manager know so they can handle the comms while the engineer works on it. Pulling in people from other teams is the engineer's call as well.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
Really a more likely scenario is something failed, a tech panicked, and started doing things that only made the problem worse

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

Docjowles posted:

I know working overseas has come up before, so sorry if this is a rehash. My wife and I have talked about moving abroad for a few years. Either Europe/UK or Australia/New Zealand. If I were to work for a large company in the US that I know for a fact has people in identical jobs in other countries, how viable is it to work there for a time and then apply for a transfer to a similar role in another country? I assume it varies widely by company and I'd ask about it during the interview process. But wondering if anyone has past experience with that.

Just curious. I have no immediate plans to do this, or leave my current job at all for that matter.

I know it's certainly easier to do it in that direction (departing the USA) than migrating to the USA. We've had devs that have wanted to move to the US and continue to work for us there, but couldn't get the 1099 (Others were successful at earlier times). They ended up working in our UK office for the US team instead!

I can't actually recall any existing staff we have making the move to NZ though, but maybe just none have applied, or it's happened and I'm just unaware of them. We do however have systems in place for new hires that happen to be overseas and want to work in NZ. It's typically a huge investment (from both employer and employee) to go to those lengths however, so it's only done if we believe it's worth it.


With regards to your logstash setup, how much time and effort did you spend on setting it up? How much do you foresee in ongoing maintenance? We currently pay a fortune for splunk, which works really well for us, but doesn't mean we shouldn't consider other options from time to time. We do have splunk doing a lot of good things for us though, such as sending emails to our team for certain log events as they happen (the kind of stuff nagios may not be able to pick up on), and simplified dashboards that our customer support team can use to determine errors particular customers may have been getting. It's fantastic, but we pay an appropriate price for it.

I always envisioned my splunk use as more Minority Report though, scrubbing out irrelevant background noise to find the killer :c00l:

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

evol262 posted:

I always thought NZ would be one of the easiest places to go, since they want developers and engineers so badly. When I visited, they gave me a packet on immigrating in case I was ever interested, because NZ neeeeddds computer people.

It is, I'm just not sure if any of our existing staff have decided to make that move after working in one of our overseas offices for a bit, but I'm well aware of NZ staff now working in one of our overseas offices.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
Dress chat, lots of the sales people in our company have visible tattoos

We sell to accountants.

Standards of dress change drastically depending on location so making blanket statements is going to get a lot of people to disagree with you on an international web forum :shrug:

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

22 Eargesplitten posted:


I was going to make it one of those gifs that gradualy grows from normal smiley size to huge, but I don't have time to learn to make .gifs right now.

10% raise, regular hours, 5 day weeks. 6 month contract with the possibility of conversion at the end of the contract. And I get experience administering tickets, rather than just responding to them. And it's based out of an office, rather than my car. So I can use a fridge and a microwave.

I'm really keen to hear the drama that comes from you handing in your notice.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
The "Force you to leave for 6 months" thing does have it's roots in security. Basically if someone is doing something dodgy, they have to be actively doing things to circumvent audits. Doing such a thing when you're not around for 6 months is practically impossible.

I think there were financial industry jobs where they mandate that you have to take 2-4 weeks leave in one block at some point in your job so they can go over your poo poo and make sure it all checks out.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

Dick Trauma posted:

I wanted to lick that paint job, and every bit of chrome was so pristine my eyes were being lasered.

So pristine I can tell you're still wearing a tie

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

Harry Lime posted:

I'm looking for some career progression advice. I recently left my old job where my title was junior systems engineer and moved to Australia and am currently job hunting. How important is it in the long term that my next job ditch the junior part of the title? I've got an interview for a junior project engineer role at an MSP that looks interesting but I'm worried how it will look on a CV that I've jumped from junior to junior. I held my previous title for 2 years and was doing help desk phone support for two years prior to that. Obviously this worry is all moot until I actually get an offer but I'm also wondering if I should just be weeding those postings out as I continue to hunt.

You can title your role however you like on your resume.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

BaseballPCHiker posted:

So I've been trying to jump ship from my current company the last few weeks. I havent gone full bore job search yet just kind of casually applying to jobs that have an easy application process and let me just upload my resume. Also only looking at jobs that would be a step-up for me money and responsibility wise. Got an interview at a place and seemed to impress them. Came in for a second interview and met the president and went way over my scheduled time, was there almost 3 hours touring the place and just bullshitting with everyone. They asked me for a list of references last week. I provided them with them expecting them to just call the people I had listed. Instead they sent out this questionnaire via email:


Is it just me or is this crazy? I'm worried that for legal reasons a lot of my references wont even reply? One person did and forwarded me their glowing responses but still kind of makes me a worry that I might not be dealing with as professional of an organization as I thought.

This looks like they outsourced their reference checking. Our company did the same because we started hiring way more people than our HR dept could reasonably keep up with, and reference checking takes time. However the people we outsource to make this a phone conversation, not just a "fill out this survey" suddenly appearing in someone's inbox unexpectedly. I wouldn't read too much into it, I know we ask similar questions, and it just means they actually care about making sure you're not a serial killer.

Lots of places have policies about what they're allowed to say, but not all of them do, and it never hurts to ask. This is not a red flag.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
Me filling out our change control form:

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

psydude posted:

Arista switches: good? They're substantially cheaper than Cisco for 10g access/distribution layer applications.

We have them in our production network, but it's all managed by the hosting company so I've never actually touched them. As far as I know we've never had any issues in our switching layer though :shrug:


As for the http.sys patch, we've applied it across our production network without much drama, what issues are people experiencing?

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
Also think about the lovely co-workers people complain about in this thread. They still have jobs.

You'll be fine.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

How you got into this really doesn't matter, only getting out does. You'll very quickly start to feel better once you realise you never have to see him ever again.

There is nothing stupid about you, this situation happens to a lot of people, you want to be nice and helpful, and a psychopath took advantage of you.

Once you're out, it'll feel so much better.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

Daylen Drazzi posted:

In the greatest moment of irony in the last couple years I just happened to find a job posting for a Virtualization Admin here at my current unit, and funnily enough I just happened to get my VCP5 today and am now qualified (on paper) for the position. I'm going to talk to the team lead about it, and if they let me transfer to the position it would make the third contractor I've worked for here without having moved more than a couple rows of desks. There are three advantages right off the top - 1) I would no longer be working the weekend shift; 2) I would get out of Exchange; 3) It's a long-term contract, as opposed to the 2-1/2 month countdown we've got going on right now. If I can talk $6-12k more out of them then it would make things about perfect.

Congrats on getting the cert man!

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

fluppet posted:

So does anyone have any advice on how to interview graduates since as they mostly have nothing relevant on their cv to ask them about

See how much they know about how the internet works with "you type in https://www.google.com and press enter, what happens?"

Figure out what sort of diagnostic chops they have with "someone reports a thing is giving an error, what can you do to figure it out?"

Depends a bit on what's relevant for your day to day stuff.

Also ask what side projects they do.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

Simon Numbers posted:

I hope I'm posting this in the right place. Just looking for advice in regards to career development.

I'm currently employed in a helldesk consumer role for a major teleco in Australia. Providing tech support for anything that can be found in the home and diagnosing network issues specific to end user. A lot of turning it on and off again as well as sending out field tasks to replace faulty cabling. The job in itself is boring as hell but it's experience and a paycheck (50k aud ain't bad)

What I'm currently looking at is trying to obtain a transfer to my companies international division that handles global networking(cloud/VPN/conferencing etc. ). I'd be working in a help desk role still but without being yelled at because someone's ping test was slow. They expect a sound knowledge of network topology at an ISP level and dnssec troubleshooting as well as video/phone conferencing.

This is something I'm interested in and am currently working towards my icdn 1 and tertiary qualifications in IT networking.

My biggest concern is I'm chasing into a another dead end and/or biting off more than I can chew.

Is there anyone here that's worked in a similar role?

Also has anyone had any experience working out of Hong Kong in an It role.

ISP level networking is both very niche, but also very cool, so it's a tricky one to give advice on. My first full time job was on a helpdesk for an ISP's enterprise level product, that was essentially VPNs over an already private backbone network (separate from regular internet traffic). I also shared a floor with the Global Gateway people, and had to interact with them when problems looked like they stemmed from that area. One of the best things about the job was that everyone I was speaking to was also a tech, including the customers, so you would have very reasonable conversations *most* of the time.

I spun a year in that role into a network admin role in that same ISP's test lab, and I was able to spin the systems admin elements of that job into the operations role I have today, which has very little network stuff involved.

I'll reiterate that the backbone level infrastructure that makes the whole internet work is fascinating to me, but most of the people who'll ever need that knowledge work for ISPs, and ISPs aren't always the best employers (lots of outsourcing/downsizing etc). But if you get really good at it, you can end up like Sepist and make mad stacks of cash. It's not a dead end, there will likely be progression through your current employer available, but personally I feel "network only" jobs are becoming scarcer, so don't feel like you shouldn't have some grounding in other IT related fields.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

Honest Thief posted:

So here's something that been going, I've started a new job roughly three months ago and 4 days ago one of the thousands many IT outsource consultancy companies phoned me up with a job opportunity, same old speech, and I accepted to go along with the process, mostly because, 'why not?'. So we schedule an interview at the client, which is a bit off-hand to where I live, basically at the city's outskirts; what happens is that they start questioning me about wanting to skip bail on the way to the interview with the client, something I found a bit weird since they're the ones who phoned and prodded me.

Anyways, not even two days later they threw at me a very attractive monetary proposal, and MUST have my reply in 24hrs or less. The money is good, but half of it is untaxed, payed through some weird legal loopholes, something related to an insurance company that will put money on a separate bank account to mine. They also seem on a hurry to lock the client, so basically they want to force him to decide after having me on board. All in all, I'm finding it a bit fishy, it seems very rushed and half baked, and they almost accuse me of just jumping from job to job and yet want me to do the same now.

On a long term perspective, am I prepping myself to get screwed over?

It's a scam, you'll never see the money in the bank account that's not yours (it's theirs)

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
I'm interviewing candidates currently, and I can't help but feel bad for the people that want to have the kind of role we're hiring for (Ops team for a SaaS company), but only have experience in more traditional enterprise type environments. They clearly want to have the type of job we're hiring for, but they're always going to lose out to the few people who have more relevant experience.

Sometimes I wonder how the hell I managed to get this job in the first place and feel pretty lucky.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

Tab8715 posted:

What's a traditional environment?

I'm assuming VMware cluster(SR+HA) and whatever on top.

Yeah the Internal IT that keeps the servers for one company functioning, and once it's working they're barely allowed to touch it.

edit: It probably worth mentioning that we're looking to emulate places like Netflix when it comes to our infrastructure, and there are not many companies at all doing what we're doing in this country.

NZAmoeba fucked around with this message at 08:19 on May 31, 2015

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

Tab8715 posted:

Curious, what does a Wanted Ad for position look like?

I hate the title, "Infrastructure Engineer" sounds too much like SANs and networks, both of which we operate at arms length.

https://www.xero.com/nz/about/careers/job/owJW0fwV/

quote:

At Xero, we're taking the world by storm, developing beautiful online accounting software that is accessible anytime, anywhere.

As Infrastructure Engineer, you’ll maintain the operations of all services across the Xero platform, while working with developers to help build a world leading product.

The infrastructure team manages 100’s of terabytes of customer data through world class data centre and infrastructure operations that underpin our live services. The data centre typology is geographically dispersed across NZ, AU and US regions and we run a hybrid of private and public cloud services.

We're looking for passionate people who want to be a part of an accounting software revolution. You need to be super comfortable in a fast-paced, high-growth company. As part of a global team, you will need to interact effectively with Platform Services teams in Wellington, Auckland & Denver, USA.

To be successful in this role you will need to:

Ideally a minimum of 2 years experience as an Systems/Infrastructure Engineer

Provide hands-on experience managing infrastructure in a high-availability environment

Support for all Production services & applications across the Xero platform

Be able to recommend and implement software and hardware solutions to meet business needs.

Ensure operational issues are resolved in a timely and cost-effective manner

Have operational experience of running a large wintel environment

Have a good understanding or have worked with Cloud technologies, such as AWS or Azure

Redhat Linux and/or powershell skills would also be beneficial

This role will involve on-call availability and periodic overtime. If you are ready to take on a new challenge in a fast-paced organisation where the sky is the limit we want to hear from you. Please apply online.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

Methanar posted:

When I saw this I immediately declared the project a failure and reverted all changes and eventually had it fully functional again.


Just going to highlight this as a Very Good Thing to do. Too many people get stuck with Go Fever and keep trying to persevere even when it's a lost cause, making things far worse. It's a hard call to make, but a good one. Well done.

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
Also prison isn't a place computer nerds do well in

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
Everyone knows it's pronounced "squirrel"

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf
I'm having a philosophical debate with some of my co-workers and I figured I'd ask the thread for some wider opinions.

I work in operations for a SaaS company, so uptime and performance are paramount. In my ideal world, any major performance problem/outage is a situation where everyone in the ops team, regardless of role, should drop what they're doing and see what they can do to assist with the situation. Even if that just means proving that the problem isn't anything to do with their area, or just running over to another team to make sure the problem has their attention. As such, when I do spot something going horrible wrong, I broadcast on every communications channel available to me to get as many people's attention as quickly as possible. Essentially I want the immediate attention of every person in the team, and then quickly whittle down to the people who can actually do something about it.

Other's disagree with this approach, as they consider it disruptive to what is a large team of people who all have their own projects going on. They feel notifications and alerts should be targeted to specific individuals, with a new person chosen if that original person isn't available. This causes a minimum of disruption to everyone else.

I've got two problems with this. 1) I feel an outage is worth everyone knowing about, even if they're not directly involved they can learn a lot by watching the situation, which could come in handy if it happens at some time in the future, out of hours when they're on call. Even if they can't help in a technical sense they can still help in other ways, in one recent example I needed a DBA but didn't have any in my office, so I needed any warm body in the other office to head over and grab the attention of a DBA there.

2) I feel it's faster to broadcast out and whittle down from there, instead of going one by one to different people until I can get someone to assist. During an outage every minute counts after all.

Where at an impasse where I'm not convincing them, but they're not convincing me either, so I'd love to hear other perspectives on it.

Adbot
ADBOT LOVES YOU

NZAmoeba
Feb 14, 2005

It turns out it's MAN!
Hair Elf

Tab8715 posted:

If there's an issue with whatever why should I care or be involved if it's nothing that falls into my teams technical expertise?

Granted, if it's fine if you want to help but sometimes having more people look into an issue doesn't mean it gets fixed faster. If I'm busy putting out fires I do not need other teams looking over my shoulder.

Because I find "Not my problem" to be a toxic way of thinking about things. And if you spot a problem that you know the resolution falls outside of your expertise to fix, do you not at least have a duty to make sure the people who do have that expertise are notified as quickly as possible?

Thanks Ants posted:

I think you need to have a defined path that issues get escalated through. If people assigned to points on that path aren't pulling their weight then you need to address that problem. I can't see how having everyone scrambling to fix an issue won't result in people treading on each others toes.

You wouldn't start paging the entire IT department at 3am if the on-call guy had a habit of ignoring high priority issues.

Edit: I can see the merit in letting people outside the team that are tasked with dealing with the issue know that there's an outage, but this needs to be something reasonably non-disruptive, like changing the status on plasma screen or a Slack notification or something low key.

You don't page the whole team at 3am, but I feel something that happens during business hours can with dealt with differently. I don't care which DBA I get, I just need one who happens to be sitting in front of a computer right now. If I specifically went to whoever was on call, they could be out getting lunch at the time, or in a meeting on a different floor to their laptop, or having a poo poo. Plenty of legitimate reasons for them to not necessarily being the best person right at that very moment when you have a whole pool of available people.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply