New around here? Register your SA Forums Account here!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Jabor
Jul 16, 2010

#1 Loser at SpaceChem
I've been on-call once as a developer, part of that was to figure out what exactly is the typical thing to break about our service so we can write a good playbook for handing it over to actual operations folks to continue running.

90% of our playbook was "escalate to whichever underlying backend service is failing to make sure they know about it", and the other 10% was "roll back the most recent release and let the devs know so they can look at it next business day".

To me, it doesn't seem very reasonable to kick a new system over to operations folks and expect them to automatically know what's going on when it starts failing and what to do about it. You need a playbook, and the stuff in that playbook needs to come from people with knowledge of the system you've built.

Adbot
ADBOT LOVES YOU

Jabor
Jul 16, 2010

#1 Loser at SpaceChem
Never be on call without being compensated specifically for being on call though.

Jabor
Jul 16, 2010

#1 Loser at SpaceChem

Blinkz0rz posted:

if it's code you're confident enough to deploy to production you should be confident that it'll run well enough that you can be paged for it

this is a stupid attitude too. being on-call means you're making sacrifices even if you never actually get paged

The Fool
Oct 16, 2003


just write code with no bugs bing bong so simple

Blinkz0rz
May 27, 2001

MY CONTEMPT FOR MY OWN EMPLOYEES IS ONLY MATCHED BY MY LOVE FOR TOM BRADY'S SWEATY MAGA BALLS

Jabor posted:

this is a stupid attitude too. being on-call means you're making sacrifices even if you never actually get paged

i’ve been on call in some form or another for the past 7 years and i’ve never made a single sacrifice beyond a few miserable nights where i took the day after off

i feel like a lot of folks that are taking a stand against being on call haven’t actually done it in any meaningful capacity

CPColin
Sep 9, 2003

Big ol' smile.
I've seen people at my work get contacted while they're on vacation and lol if they ever try that with me. It'll be me telling them to figure it out themselves AND taking back my vacation day.

Achmed Jones
Oct 16, 2004



eh your hobbies matter. similarly, activities where you can't easily get to a computer are out. you can't just run off the field in team sports, you can't keep your phone on you during a martial arts class (and are likely to miss it if it's in your bag or whatever), anything out of cell range is a no-go, etc.

plus there's a lot of people in yospos who drink or smoke themselves into oblivion every chance they get, which they can't do if they're on call.

both the obliteration and leaving-the-house leisure activities are a bigger ask to avoid on weekends for obvious reasons.

Achmed Jones
Oct 16, 2004



blink what kind of page-to-work-begins sla are you talkin? my work has two levels iirc, 5min and 1hour. 5min is absolutely a huge impact to peoples lives. 1hour less so, but the above still mostly applies

my homie dhall
Dec 9, 2010

honey, oh please, it's just a machine
playbooks are bullshit too, if you have scenarios you can write a playbook for, you can automate responses to them

The Fool
Oct 16, 2003


well that's just wrong

Blinkz0rz
May 27, 2001

MY CONTEMPT FOR MY OWN EMPLOYEES IS ONLY MATCHED BY MY LOVE FOR TOM BRADY'S SWEATY MAGA BALLS

Achmed Jones posted:

blink what kind of page-to-work-begins sla are you talkin? my work has two levels iirc, 5min and 1hour. 5min is absolutely a huge impact to peoples lives. 1hour less so, but the above still mostly applies

we don’t have a page-to-work sla. for low sev we alert during the business day and high sev is 24/7

we escalate to tier 2 after 30 minutes and then to eng management 30 minutes after that. we don’t let it get to eng management.

Jabor
Jul 16, 2010

#1 Loser at SpaceChem

Blinkz0rz posted:

i’ve been on call in some form or another for the past 7 years and i’ve never made a single sacrifice beyond a few miserable nights where i took the day after off

i feel like a lot of folks that are taking a stand against being on call haven’t actually done it in any meaningful capacity

Maybe you've never gone camping for a weekend somewhere there's no cell reception, but a lot of people enjoy doing stuff like that and it's fundamentally incompatible with being on-call no matter how infrequently pages actually happen.

You can't go to a movie and turn your phone off for 2 hours either, given your claimed SLOs.

PokeJoe
Aug 24, 2004

I like to go backpacking, hiking, and camping many hours away from civilization. I'm absolutely not gonna give that up to be available to fix source files in 30 mins or less

Blinkz0rz
May 27, 2001

MY CONTEMPT FOR MY OWN EMPLOYEES IS ONLY MATCHED BY MY LOVE FOR TOM BRADY'S SWEATY MAGA BALLS

PokeJoe posted:

I like to go backpacking, hiking, and camping many hours away from civilization. I'm absolutely not gonna give that up to be available to fix source files in 30 mins or less

that’s cool there are folks on my team that are into that stuff too

they just pick weekends when they’re not on-call

PokeJoe
Aug 24, 2004

there's only a few months a year here to do it so I'm not gonna do it only one weekend a month or whatever. The outdoors are my only hobby during the summers am I gonna take 3 months off of on call shifts? it's fine you're cool with on call shifts but the idea that every dev should be is a lovely one

JawnV6
Jul 4, 2004

So hot ...
at my first job the absolute breakneck fastest pace a change could get from an engineers computer into a customers hands was at least 4 months

Blinkz0rz
May 27, 2001

MY CONTEMPT FOR MY OWN EMPLOYEES IS ONLY MATCHED BY MY LOVE FOR TOM BRADY'S SWEATY MAGA BALLS

PokeJoe posted:

there's only a few months a year here to do it so I'm not gonna do it only one weekend a month or whatever. The outdoors are my only hobby during the summers am I gonna take 3 months off of on call shifts? it's fine you're cool with on call shifts but the idea that every dev should be is a lovely one

sounds like it doesn't work for you so don't work for saas companies that provide 24/7 services

in general though i don't think it's unreasonable to expect the folks who write code to be responsible for the operational quality of their code. if you're confident in qa and test processes then you should be confident enough to be paged for it. "you build it, you own it" feels like table stakes for modern orgs but idk

Achmed Jones
Oct 16, 2004



yeah sorry blink, but "give up your preferred leisure activities one weekend out of every N" isn't gonna fly for a lot of people unless the sentence ends with "...and you will be paid for it, even if you don't catch a page." that's how oncall works where i work, and i wouldn't go back to what you're describing because it sucks real bad. ive been contacted during my son's birthday lunch. ive been contacted when i was in the middle of evacuating from a natural disaster. never doing that again unless there's really fat stacks attached (and it's still gonna be a hard sell given i already make faang money and like my current oncall schedule of "never")

PokeJoe
Aug 24, 2004

I don't own it, I can't take it with me when I leave. Push back against these stupid terms and rules. The comany paid me to make it for them to own.

Jabor
Jul 16, 2010

#1 Loser at SpaceChem

Blinkz0rz posted:

that’s cool there are folks on my team that are into that stuff too

they just pick weekends when they’re not on-call

i know you claim you learned a lot from your early managerial fuckups but it honestly sounds like you really haven't, lmao

Achmed Jones
Oct 16, 2004



maybe googles not a modern org but "pay your oncallers" is table stakes imo. being oncall can be fine. only being compensated for pages, and then only with time in lieu, is some bullshit. that you've turned it into a point of pride or a marker of modernity is kinda strange. you aren't getting pushback for the rotation, you're getting it because you do it for free

PokeJoe
Aug 24, 2004

As soon as the code is merged it's no longer "mine"

Blinkz0rz
May 27, 2001

MY CONTEMPT FOR MY OWN EMPLOYEES IS ONLY MATCHED BY MY LOVE FOR TOM BRADY'S SWEATY MAGA BALLS

Achmed Jones posted:

yeah sorry blink, but "give up your preferred leisure activities one weekend out of every N" isn't gonna fly for a lot of people unless the sentence ends with "...and you will be paid for it, even if you don't catch a page." that's how oncall works where i work, and i wouldn't go back to what you're describing because it sucks real bad. ive been contacted during my son's birthday lunch. ive been contacted when i was in the middle of evacuating from a natural disaster. never doing that again unless there's really fat stacks attached (and it's still gonna be a hard sell given i already make faang money and like my current oncall schedule of "never")

yeah i’ll never be on call with a sla without being explicitly compensated for it. even if my hobbies aren’t being off the grid i still don’t want to work on my own time.

i just recognize that throwing my hands up and saying “not my problem” just makes it someone else’s problem and now we’re back to a world where dev and ops hate each other and no one knows anything about how software is created or run. i’ve worked in orgs like that before and they’re absolutely miserable.

PokeJoe
Aug 24, 2004

If you need 24/7 support the secret is as old as time: pay someone to work support 2nd and 3rd shift

Archduke Frantz Fanon
Sep 7, 2004

PokeJoe posted:

If you need 24/7 support the secret is as old as time: pay someone to work support 2nd and 3rd shift

:hmmyes:

devs arent support. if you need the devs to come in off hours to fix something it should be with a fully developed set of requirements and a plan to test and deploy it. otherwise its just gonna more tech debt and the cycle continues until your pm is burnt out and all your devs leave for more boring jobs.

ideally poo poo should be set up in a way that support and ops, fully paid to be graveyard shifters, can handle it until normal business hours

places dont because why pay for more people when programmers can just do everything?

PokeJoe
Aug 24, 2004

Archduke Frantz Fanon posted:

:hmmyes:

devs arent support. if you need the devs to come in off hours to fix something it should be with a fully developed set of requirements and a plan to test and deploy it. otherwise its just gonna more tech debt and the cycle continues until your pm is burnt out and all your devs leave for more boring jobs.

ideally poo poo should be set up in a way that support and ops, fully paid to be graveyard shifters, can handle it until normal business hours

places dont because why pay for more people when programmers can just do everything?

loving bingo

Achmed Jones
Oct 16, 2004



wait how is "it escalates to management after an hour and they get woken up, so we never let it get that far" not basically an sla?

i generally don't mind devs being ops. esp at the sort of shops blink is talking about, they're often best equipped to fix the issue and the build pipeline can support that kind of thing, and being on the hook really does ameliorate "over the wall" nonsense to some degree. but they need to get paid for the mental space they rent out to the company, and hiring really should support distributed time zones or alternate shift schedules to make life easier on everybody (but this is just another way of saying "no free work")

DELETE CASCADE
Oct 25, 2017

i haven't washed my penis since i jerked it to a phtotograph of george w. bush in 2003
if you're a decent dev and not helpdesk, you can make it a give and take when you work for one of those companies. "sure, i'll be constantly on call for all customer issues, without expectation of additional pay. but i do not promise sobriety outside of work hours, and i'll be taking those calls from maui along with the rest of my fully remote role, thank you very much"

MononcQc
May 29, 2007

not all call rotations are equal and I’ve had lovely ones with 5 alarms a night and boring ones with alerts a year. the first one burns you out, the latter stresses you out because you’re out of practice.

My prior one was at a food delivery app so peak times were always dinner times and that’s when you’d get paged. My current one mostly pages during business hours so on-call is operational time and agency to fix and improve poo poo.

there’s legitimately ways of designing oncall schedules that aren’t burnout machines.

my current role had some tricky alerts so I said “hey this isn’t actionable and can’t be fixed in the scope of oncall so I’m turning all alerting off for these things, now we gotta talk and decide what to do about it” and we sat down with product and support to figure out how to make things better without burning people out.

I find on call a lot more bearable when it comes with autonomy and agency and is seen as a critical part of software development rather than being some sort of poo poo job for second tier workers. Sort of the same way tests are actually nicer when they’re an expected part of the workflow rather than something you ship off to another department in a quasi adversarial manner.

DELETE CASCADE
Oct 25, 2017

i haven't washed my penis since i jerked it to a phtotograph of george w. bush in 2003
i do wonder to what extent those poo poo job second tier workers are also secretly longing for autonomy and agency, vs how many just want to pull tickets off the pile until quittin time (which is perfectly fine with me)

maybe they would be happier with the latter if a bit of the former was included? oh well, the world will never know i guess

Qtotonibudinibudet
Nov 7, 2011



Omich poluyobok, skazhi ty narkoman? ya prosto tozhe gde to tam zhivu, mogli by vmeste uyobyvat' narkotiki

nudgenudgetilt posted:

Kuvo posted:

"i have faith $COMPANY can make a competitive offer based on my skillset and experience"

this makes you sound like an asshat. don't ever say this.

either just say "i can't see the numbers working out, thanks though", or "i'm pretty happy with my current situation, but might reconsider for $fig"

the person asking salary questions is (at all but the tiniest of startups) someone in HR. you will not interact with them after getting hired, and it is literally their job to give you the lowest offer you will take. you can reasonably just say "market rate; if you can provide a range i can tell you whether i'd switch for that to not waste your time"

in either case, unless op has moved recently, as of 2018: California Code, Labor Code - LAB § 432.3

quote:

(c) An employer, upon reasonable request, shall provide the pay scale for a position to an applicant applying for employment. For purposes of this section, “pay scale” means a salary or hourly wage range. For purposes of this section, “reasonable request” means a request made after an applicant has completed an initial interview with the employer.

Raluek
Nov 3, 2006

WUT.

CMYK BLYAT! posted:

in either case, unless op has moved recently, as of 2018: California Code, Labor Code - LAB § 432.3

this is correct

i would phrase it something like "i am pretty comfortable where i am now, and am not really looking to make a change, but i could be motivated by the right offer. what is your range?" once you feel like you understand the role well enough to start talking about comp

4lokos basilisk
Jul 17, 2008


re: on call. thanks for the different points made

im doing some weekend on call at my current place which is paid, but its definitely not the same amount than my regular hourly rate. this also involves reacting to pages at 3am

i generally agree with being responsible for my work, but:
* for 24/7 support there ought to be a person somewhere around the globe able to react
* the above means weekend on call compensated generously
* i will not be held responsible for something the team or other teams decided to ship without adequate qa
* i will want to be able to do my work to at least my own quality standards

it really boils down to this: if the thing my team is responsible for must have high sla, then the everyday processes in the team and the company as a whole should be aligned with this.

telling me this in an interview is ehh because at that point i have no idea what the standards and quality culture exist in the company

i guess i have some questions here :)

Blinkz0rz
May 27, 2001

MY CONTEMPT FOR MY OWN EMPLOYEES IS ONLY MATCHED BY MY LOVE FOR TOM BRADY'S SWEATY MAGA BALLS

Achmed Jones posted:

wait how is "it escalates to management after an hour and they get woken up, so we never let it get that far" not basically an sla?

i generally don't mind devs being ops. esp at the sort of shops blink is talking about, they're often best equipped to fix the issue and the build pipeline can support that kind of thing, and being on the hook really does ameliorate "over the wall" nonsense to some degree. but they need to get paid for the mental space they rent out to the company, and hiring really should support distributed time zones or alternate shift schedules to make life easier on everybody (but this is just another way of saying "no free work")

it’s time to ack not time to resolve

i get that’s sort of an implied sla but idk it doesn’t really bother me that much. it’s not like someone gets fired if it gets to management or anything. it just becomes a process improvement thing

Jabor
Jul 16, 2010

#1 Loser at SpaceChem
time-to-ack is in fact a very common slo to define for someone carrying a pager

(it's not an sla because the penalty for breaking it is just that someone gets annoyed and maybe has a word to you about not meeting your objectives, as opposed to any sort of contractual penalty)

champagne posting
Apr 5, 2006

YOU ARE A BRAIN
IN A BUNKER


I kinda wanna apply for this job is this job: https://www.linkedin.com/jobs/view/3075072299/ , talk me out of it

CarForumPoster
Jun 26, 2013

champagne posting posted:

I kinda wanna apply for this job is this job: https://www.linkedin.com/jobs/view/3075072299/ , talk me out of it

If you're an okay PM and technical manager seems like a decent job?

champagne posting
Apr 5, 2006

YOU ARE A BRAIN
IN A BUNKER


CarForumPoster posted:

If you're an okay PM and technical manager seems like a decent job?

I am not yet either of those things

KirbyKhan
Mar 20, 2009

I purchased and 100%'d Huniecam so quickly that I still get friends shaming me over it. And I deserve to be shamed. But I did not praise the game. I called it fucked up. That's how fucked up it is: it's up there with Princess Maker as far as poster body counts.
Soiled Meat

Jabor posted:

this is a stupid attitude too. being on-call means you're making sacrifices even if you never actually get paged

I was on call without additional pay while I was in the military, where I was actually owned by my employers. This was to keep things running after some Scandanavian volcano threw ash in the air that degraded a fleet of aircraft. This is my personal bar for urgent on-call activity, if it doesn't pass that bar then fuuuuuuuuuuck that pay me and give me a phone.

my homie dhall posted:

playbooks are bullshit too, if you have scenarios you can write a playbook for, you can automate responses to them

Playbooks are meant to be read by someone you've never met, to be stored in a drawer nobody opens, during an hour no one should be in. They are invaluable to operations that are 24/7, have kafka-esq turnover, and way more people than a single brain can keep phone numbers for.

Adbot
ADBOT LOVES YOU

Achmed Jones
Oct 16, 2004



oh yeah i was talking internal sla, not contractually enforced ones

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply