|
I've been on-call once as a developer, part of that was to figure out what exactly is the typical thing to break about our service so we can write a good playbook for handing it over to actual operations folks to continue running. 90% of our playbook was "escalate to whichever underlying backend service is failing to make sure they know about it", and the other 10% was "roll back the most recent release and let the devs know so they can look at it next business day". To me, it doesn't seem very reasonable to kick a new system over to operations folks and expect them to automatically know what's going on when it starts failing and what to do about it. You need a playbook, and the stuff in that playbook needs to come from people with knowledge of the system you've built.
|
![]() |
|
![]()
|
# ? Feb 11, 2025 01:30 |
|
Never be on call without being compensated specifically for being on call though.
|
![]() |
|
Blinkz0rz posted:if it's code you're confident enough to deploy to production you should be confident that it'll run well enough that you can be paged for it this is a stupid attitude too. being on-call means you're making sacrifices even if you never actually get paged
|
![]() |
|
just write code with no bugs bing bong so simple
|
![]() |
|
Jabor posted:this is a stupid attitude too. being on-call means you're making sacrifices even if you never actually get paged i’ve been on call in some form or another for the past 7 years and i’ve never made a single sacrifice beyond a few miserable nights where i took the day after off i feel like a lot of folks that are taking a stand against being on call haven’t actually done it in any meaningful capacity
|
![]() |
|
I've seen people at my work get contacted while they're on vacation and lol if they ever try that with me. It'll be me telling them to figure it out themselves AND taking back my vacation day.
|
![]() |
|
eh your hobbies matter. similarly, activities where you can't easily get to a computer are out. you can't just run off the field in team sports, you can't keep your phone on you during a martial arts class (and are likely to miss it if it's in your bag or whatever), anything out of cell range is a no-go, etc. plus there's a lot of people in yospos who drink or smoke themselves into oblivion every chance they get, which they can't do if they're on call. both the obliteration and leaving-the-house leisure activities are a bigger ask to avoid on weekends for obvious reasons.
|
![]() |
|
blink what kind of page-to-work-begins sla are you talkin? my work has two levels iirc, 5min and 1hour. 5min is absolutely a huge impact to peoples lives. 1hour less so, but the above still mostly applies
|
![]() |
|
playbooks are bullshit too, if you have scenarios you can write a playbook for, you can automate responses to them
|
![]() |
|
well that's just wrong
|
![]() |
|
Achmed Jones posted:blink what kind of page-to-work-begins sla are you talkin? my work has two levels iirc, 5min and 1hour. 5min is absolutely a huge impact to peoples lives. 1hour less so, but the above still mostly applies we don’t have a page-to-work sla. for low sev we alert during the business day and high sev is 24/7 we escalate to tier 2 after 30 minutes and then to eng management 30 minutes after that. we don’t let it get to eng management.
|
![]() |
|
Blinkz0rz posted:i’ve been on call in some form or another for the past 7 years and i’ve never made a single sacrifice beyond a few miserable nights where i took the day after off Maybe you've never gone camping for a weekend somewhere there's no cell reception, but a lot of people enjoy doing stuff like that and it's fundamentally incompatible with being on-call no matter how infrequently pages actually happen. You can't go to a movie and turn your phone off for 2 hours either, given your claimed SLOs.
|
![]() |
|
I like to go backpacking, hiking, and camping many hours away from civilization. I'm absolutely not gonna give that up to be available to fix source files in 30 mins or less
|
![]() |
|
PokeJoe posted:I like to go backpacking, hiking, and camping many hours away from civilization. I'm absolutely not gonna give that up to be available to fix source files in 30 mins or less that’s cool there are folks on my team that are into that stuff too they just pick weekends when they’re not on-call
|
![]() |
|
there's only a few months a year here to do it so I'm not gonna do it only one weekend a month or whatever. The outdoors are my only hobby during the summers am I gonna take 3 months off of on call shifts? it's fine you're cool with on call shifts but the idea that every dev should be is a lovely one
|
![]() |
|
at my first job the absolute breakneck fastest pace a change could get from an engineers computer into a customers hands was at least 4 months
|
![]() |
|
PokeJoe posted:there's only a few months a year here to do it so I'm not gonna do it only one weekend a month or whatever. The outdoors are my only hobby during the summers am I gonna take 3 months off of on call shifts? it's fine you're cool with on call shifts but the idea that every dev should be is a lovely one sounds like it doesn't work for you so don't work for saas companies that provide 24/7 services in general though i don't think it's unreasonable to expect the folks who write code to be responsible for the operational quality of their code. if you're confident in qa and test processes then you should be confident enough to be paged for it. "you build it, you own it" feels like table stakes for modern orgs but idk
|
![]() |
|
yeah sorry blink, but "give up your preferred leisure activities one weekend out of every N" isn't gonna fly for a lot of people unless the sentence ends with "...and you will be paid for it, even if you don't catch a page." that's how oncall works where i work, and i wouldn't go back to what you're describing because it sucks real bad. ive been contacted during my son's birthday lunch. ive been contacted when i was in the middle of evacuating from a natural disaster. never doing that again unless there's really fat stacks attached (and it's still gonna be a hard sell given i already make faang money and like my current oncall schedule of "never")
|
![]() |
|
I don't own it, I can't take it with me when I leave. Push back against these stupid terms and rules. The comany paid me to make it for them to own.
|
![]() |
|
Blinkz0rz posted:that’s cool there are folks on my team that are into that stuff too i know you claim you learned a lot from your early managerial fuckups but it honestly sounds like you really haven't, lmao
|
![]() |
|
maybe googles not a modern org but "pay your oncallers" is table stakes imo. being oncall can be fine. only being compensated for pages, and then only with time in lieu, is some bullshit. that you've turned it into a point of pride or a marker of modernity is kinda strange. you aren't getting pushback for the rotation, you're getting it because you do it for free
|
![]() |
|
As soon as the code is merged it's no longer "mine"
|
![]() |
|
Achmed Jones posted:yeah sorry blink, but "give up your preferred leisure activities one weekend out of every N" isn't gonna fly for a lot of people unless the sentence ends with "...and you will be paid for it, even if you don't catch a page." that's how oncall works where i work, and i wouldn't go back to what you're describing because it sucks real bad. ive been contacted during my son's birthday lunch. ive been contacted when i was in the middle of evacuating from a natural disaster. never doing that again unless there's really fat stacks attached (and it's still gonna be a hard sell given i already make faang money and like my current oncall schedule of "never") yeah i’ll never be on call with a sla without being explicitly compensated for it. even if my hobbies aren’t being off the grid i still don’t want to work on my own time. i just recognize that throwing my hands up and saying “not my problem” just makes it someone else’s problem and now we’re back to a world where dev and ops hate each other and no one knows anything about how software is created or run. i’ve worked in orgs like that before and they’re absolutely miserable.
|
![]() |
|
If you need 24/7 support the secret is as old as time: pay someone to work support 2nd and 3rd shift
|
![]() |
|
PokeJoe posted:If you need 24/7 support the secret is as old as time: pay someone to work support 2nd and 3rd shift ![]() devs arent support. if you need the devs to come in off hours to fix something it should be with a fully developed set of requirements and a plan to test and deploy it. otherwise its just gonna more tech debt and the cycle continues until your pm is burnt out and all your devs leave for more boring jobs. ideally poo poo should be set up in a way that support and ops, fully paid to be graveyard shifters, can handle it until normal business hours places dont because why pay for more people when programmers can just do everything?
|
![]() |
|
Archduke Frantz Fanon posted:
loving bingo
|
![]() |
|
wait how is "it escalates to management after an hour and they get woken up, so we never let it get that far" not basically an sla? i generally don't mind devs being ops. esp at the sort of shops blink is talking about, they're often best equipped to fix the issue and the build pipeline can support that kind of thing, and being on the hook really does ameliorate "over the wall" nonsense to some degree. but they need to get paid for the mental space they rent out to the company, and hiring really should support distributed time zones or alternate shift schedules to make life easier on everybody (but this is just another way of saying "no free work")
|
![]() |
|
if you're a decent dev and not helpdesk, you can make it a give and take when you work for one of those companies. "sure, i'll be constantly on call for all customer issues, without expectation of additional pay. but i do not promise sobriety outside of work hours, and i'll be taking those calls from maui along with the rest of my fully remote role, thank you very much"
|
![]() |
|
not all call rotations are equal and I’ve had lovely ones with 5 alarms a night and boring ones with alerts a year. the first one burns you out, the latter stresses you out because you’re out of practice. My prior one was at a food delivery app so peak times were always dinner times and that’s when you’d get paged. My current one mostly pages during business hours so on-call is operational time and agency to fix and improve poo poo. there’s legitimately ways of designing oncall schedules that aren’t burnout machines. my current role had some tricky alerts so I said “hey this isn’t actionable and can’t be fixed in the scope of oncall so I’m turning all alerting off for these things, now we gotta talk and decide what to do about it” and we sat down with product and support to figure out how to make things better without burning people out. I find on call a lot more bearable when it comes with autonomy and agency and is seen as a critical part of software development rather than being some sort of poo poo job for second tier workers. Sort of the same way tests are actually nicer when they’re an expected part of the workflow rather than something you ship off to another department in a quasi adversarial manner.
|
![]() |
|
i do wonder to what extent those poo poo job second tier workers are also secretly longing for autonomy and agency, vs how many just want to pull tickets off the pile until quittin time (which is perfectly fine with me) maybe they would be happier with the latter if a bit of the former was included? oh well, the world will never know i guess
|
![]() |
|
nudgenudgetilt posted:
the person asking salary questions is (at all but the tiniest of startups) someone in HR. you will not interact with them after getting hired, and it is literally their job to give you the lowest offer you will take. you can reasonably just say "market rate; if you can provide a range i can tell you whether i'd switch for that to not waste your time" in either case, unless op has moved recently, as of 2018: California Code, Labor Code - LAB § 432.3 quote:(c) An employer, upon reasonable request, shall provide the pay scale for a position to an applicant applying for employment. For purposes of this section, “pay scale” means a salary or hourly wage range. For purposes of this section, “reasonable request” means a request made after an applicant has completed an initial interview with the employer.
|
![]() |
|
CMYK BLYAT! posted:in either case, unless op has moved recently, as of 2018: California Code, Labor Code - LAB § 432.3 this is correct i would phrase it something like "i am pretty comfortable where i am now, and am not really looking to make a change, but i could be motivated by the right offer. what is your range?" once you feel like you understand the role well enough to start talking about comp
|
![]() |
|
re: on call. thanks for the different points made im doing some weekend on call at my current place which is paid, but its definitely not the same amount than my regular hourly rate. this also involves reacting to pages at 3am i generally agree with being responsible for my work, but: * for 24/7 support there ought to be a person somewhere around the globe able to react * the above means weekend on call compensated generously * i will not be held responsible for something the team or other teams decided to ship without adequate qa * i will want to be able to do my work to at least my own quality standards it really boils down to this: if the thing my team is responsible for must have high sla, then the everyday processes in the team and the company as a whole should be aligned with this. telling me this in an interview is ehh because at that point i have no idea what the standards and quality culture exist in the company i guess i have some questions here ![]()
|
![]() |
|
Achmed Jones posted:wait how is "it escalates to management after an hour and they get woken up, so we never let it get that far" not basically an sla? it’s time to ack not time to resolve i get that’s sort of an implied sla but idk it doesn’t really bother me that much. it’s not like someone gets fired if it gets to management or anything. it just becomes a process improvement thing
|
![]() |
|
time-to-ack is in fact a very common slo to define for someone carrying a pager (it's not an sla because the penalty for breaking it is just that someone gets annoyed and maybe has a word to you about not meeting your objectives, as opposed to any sort of contractual penalty)
|
![]() |
|
I kinda wanna apply for this job is this job: https://www.linkedin.com/jobs/view/3075072299/ , talk me out of it
|
![]() |
|
champagne posting posted:I kinda wanna apply for this job is this job: https://www.linkedin.com/jobs/view/3075072299/ , talk me out of it If you're an okay PM and technical manager seems like a decent job?
|
![]() |
|
CarForumPoster posted:If you're an okay PM and technical manager seems like a decent job? I am not yet either of those things
|
![]() |
|
Jabor posted:this is a stupid attitude too. being on-call means you're making sacrifices even if you never actually get paged I was on call without additional pay while I was in the military, where I was actually owned by my employers. This was to keep things running after some Scandanavian volcano threw ash in the air that degraded a fleet of aircraft. This is my personal bar for urgent on-call activity, if it doesn't pass that bar then fuuuuuuuuuuck that pay me and give me a phone. my homie dhall posted:playbooks are bullshit too, if you have scenarios you can write a playbook for, you can automate responses to them Playbooks are meant to be read by someone you've never met, to be stored in a drawer nobody opens, during an hour no one should be in. They are invaluable to operations that are 24/7, have kafka-esq turnover, and way more people than a single brain can keep phone numbers for.
|
![]() |
|
![]()
|
# ? Feb 11, 2025 01:30 |
|
oh yeah i was talking internal sla, not contractually enforced ones
|
![]() |