Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
12 rats tied together
Sep 7, 2006

For EKS we had the whole thing in Ansible. The cluster creator supplies like a 10 line config dictionary and runs the playbook, is greeted with a production EKS cluster with all of the assumed services, integrations, namespaces, user access config, etc in like 15 minutes.

Deploying a business application to it was as simple as appending a role to the playbook. Most of the work was in roles and role dependencies, which handled other providers like Azure AKS+ARM, so the entire k8s code footprint was like 94% shared code with some parameters.

It's a fantastic workflow tbh

Adbot
ADBOT LOVES YOU

Methanar
Sep 26, 2013

by the sex ghost

freeasinbeer posted:

So you use it to bootstrap nodes? Or are you doing using it for arbitrary kubernetes deployments? If so your one of the few I’ve heard of that use it that way.

Across multiple orgs the total amount of cm code I’ve seen is weighted heavily away from tooling like ansible or puppet as compared to what it used to be, in particular any app configs that existed.

Ansible builds AMIs with everything that can be pre-installed initially. Anything apt including the kubelet/kubectl binaries, some generated config files, apparmor profiles, PSP definitions, pre-pulls any docker images we'll need.

Then Terraform we set some 3 bash scripts into s3 namespaced for the cluster we're building: init.sh, leader.sh, and join.sh

userdata contains some logic that starts a few things like cloudwatch before executing init.sh

init.sh is basically just a leader election process that picks one of the booting master nodes to be the leader, if master leader go to leader.sh, else go to join.

leader.sh does a bunch of 1 time config stuff like run kubeadm init, install cilium, install some CRD definitions, PSPs, certain rbac, a bunch of stuff like that. the kubeadm join tokens are dumped to s3. Most of the extra install stuff here is just applying config files that were laid down by ansible.

join.sh loops until it has a valid API server to speak to behind the lb and then once it does it'll attempt to kubeadm join off the token put into s3.

At this point you have a valid cluster. Ongoing maintenance is handled by the same Ansible that initially built the AMI. stuff like, upgrading k8s itself, upgrading manifests, changing the kubeadm config files to change variables set to some component


Its nice that the ansible overlaps between bootstrapping initially and upgrades. Just make your changes in ansible, apply it and then cut a new AMI and future nodes that come up will already be in the correct upgraded state.

Methanar
Sep 26, 2013

by the sex ghost
We didn't use EKS because of three reasons.

We used k8s before EKS existed.
The org wanted to have more technical expertise in Kubernetes and building your own stack is a good way to do it

The org really doesn't want to vendorlock to AWS. It was known early on that k8s was going to be the abstraction layer we build against opposed to AWS when it comes to applications. Our workloads are at a sufficient scale that being able to move bits and pieces to DCs to run baseload on our own hardware is worth it. Therefore we needed to have our own scheme anyway to build and own k8s in DCs; and maybe GCP in case we ever want to pull a netflix and build a GCP version of the product for the sole purpose of negotiating with AWS

12 rats tied together
Sep 7, 2006

Methanar posted:

Its nice that the ansible overlaps between bootstrapping initially and upgrades. Just make your changes in ansible, apply it and then cut a new AMI and future nodes that come up will already be in the correct upgraded state.

It's hard to put into words how nice this is when you have an org that has a fully developed set of ansible muscle memory, it's a single pane of glass for any set of ordered actions that can be described in a repeatable way. You barely even need documentation, you can just pull up git history and get the full view into how any process has ever looked, who changed it, when, and why.

For us, since we were just using the EKS AMIs, this process was basically: append a new ASG to the cloudformation stack for a particular AWS::EKS::Cluster. The new ASG has the latest EKS AMI, next ansible grabs all of the old nodes (trivial to discover through cloudformation outputs -- they're on the old asg(s)) and steps through and cordons them. When the process is done we merge another PR to remove the old ASG(s) from the stack. Updating a cluster is 2 pull requests and like 30 minutes of work, depending on node availability.

Cloudformation is very good at this and will reserve/rollback to your last known working state, additionally, Cloudformation has excellent documentation that describes exactly what happens when you change any property of any resource. Destructive changes are orchestrated through cfn changesets, which is just terraform plan but without the lying.

At any point in this process ansible can pause and run health checks using assert/until, so you can be like "drain 25% of the old nodes, wait for statefulset whatever to have its full set of scheduled pods again, wait for messages processed/sec to no longer be in alert, repeat until all nodes are done". The configuration for running that type of deployment is 0 code and all standard ansible features.

Methanar
Sep 26, 2013

by the sex ghost

12 rats tied together posted:

wait for messages processed/sec to no longer be in alert

Neat, how do you handle this?

12 rats tied together
Sep 7, 2006

Methanar posted:

Neat, how do you handle this?

Since I was perpetually rushing at this place, I cheated by using the prometheus http api, replicating the alert query, and checking whether or not the result was 0 or 1. You can manually invoke the url lookup inside an "until" block:
code:
- name: do something, or whatever
  module_name:
    param: value
  until: lookup('url', 'http://prometheus.whatever/query=<some nasty stuff>').result == 0
If you don't have a specific task already you can use the uri module, register the result, and then until on that result (IIRC, it's been a while).

If I had more time to spend on it I would have written a lookup plugin for it, since I had to do a bunch of url encoding for some of the queries. You could probably also hit AlertManager directly, but we had some internal politicking that prevented me from doing this.

More broadly you can use this pattern with any alert service that will give you info over HTTP. The biggest user of this was kafka clusters, where we wanted to restart services on each node, but we wanted to wait for any/all under replicated partition alerts to disappear before moving onto the next node. In ansible this is of course pretty simple, apply serial=1 on your play and set your alert check as a pre_task.

Methanar
Sep 26, 2013

by the sex ghost

12 rats tied together posted:

Since I was perpetually rushing at this place, I cheated by using the prometheus http api, replicating the alert query, and checking whether or not the result was 0 or 1. You can manually invoke the url lookup inside an "until" block:
code:
- name: do something, or whatever
  module_name:
    param: value
  until: lookup('url', 'http://prometheus.whatever/query=<some nasty stuff>').result == 0
If you don't have a specific task already you can use the uri module, register the result, and then until on that result (IIRC, it's been a while).

If I had more time to spend on it I would have written a lookup plugin for it, since I had to do a bunch of url encoding for some of the queries. You could probably also hit AlertManager directly, but we had some internal politicking that prevented me from doing this.

More broadly you can use this pattern with any alert service that will give you info over HTTP. The biggest user of this was kafka clusters, where we wanted to restart services on each node, but we wanted to wait for any/all under replicated partition alerts to disappear before moving onto the next node. In ansible this is of course pretty simple, apply serial=1 on your play and set your alert check as a pre_task.

That's awesome.

We've been looking at doing something similar with tying auto canary analysis into Spinnaker for our releases for a while now.

I have at least one app/developer that I set up who's doing hpa autoscaling with the results of arbitrary prometheus queries for some internal app metric using keda https://keda.sh/docs/1.4/scalers/prometheus/

Methanar fucked around with this message at 20:00 on Oct 29, 2020

guppy
Sep 21, 2004

sting like a byob
As I've mentioned in the past, I work in a group that doesn't know how to automate things and doesn't want to learn, and in some cases fears it as a concept. I am not a great coder, but I've been working really hard to build those skills. Today I wrote a script that is going to turn 30-60 minutes of work per site into about 50 seconds per site, and is going to be useful across a couple hundred sites. (It will probably be faster once I learn threading.) It took a few hours to write and I have some cleanup to do to make it look nice, but it seems to be working well. I am torn between the desire to show off share it with my team and the desire for people to think this still takes a long time.

This is Python and it's not a platform supported by netmiko, so I've had to brave the jungles of Paramiko, which I have zero prior experience with. I found some examples online, including one that someone got working for this specific platform, and I was able to adapt them into what I needed. I did have kind of a weird experience where the functionality worked for a given device but when iterating over a list of device IPs, it wouldn't finish the first one and move on. Except that once or twice, it appeared to do two before hanging. I thought I had some kind of flow control problem, but then I can't explain why once in a while it would do more than one. Other times it didn't even try to execute the first one. Very weird.

The example that I found for this platform used pprint to display the stdout buffer, which I don't usually bother with, but I got a not-very-helpful error message when I tried to call it, so I had just turned them into regular print statements. I realized a bit later that I had imported the entire pprint package (vs "from pprint import pprint") but was calling pprint without the package name, hence the error. I changed the import statement and for some reason that seems to have made it work. Maybe the raw text in the buffer is causing some kind of escaping problem?

This seems like a weird edge case, too, where you can't just use the apparently-standard "stdin, stdout, stderr = SSHClient.exec_command('command')" syntax and instead I had to manually manage the data channels. (I think that possibly adding the get_pty flag might have made this work. I don't really understand Paramiko yet!)

I am given to understand that if I put my SSH session inside a with block (I think this is called a context manager?) then I don't have to manually close it and it will be automatically closed when the block finishes. Is that right? I manually closed it anyway just in case because I was worried that maybe it stopped working because I was leaving a ton of open connections, but it didn't seem to make any difference.

Coffee Jones
Jul 4, 2004

16 bit? Back when we was kids we only got a single bit on Christmas, as a treat
And we had to share it!
We have a great system to track our certs, it warns you when they expire, let’s you know how they’re being used, which team owns them ... except one wasn’t entered into our system and the guy that did it is now working elsewhere.

Guess what happened today.


Anyway, got to meet the CTO, the VP of product, several SREs, the escalation engineers who follow up with customers...

Coffee Jones fucked around with this message at 07:28 on Oct 30, 2020

SEKCobra
Feb 28, 2011

Hi
:saddowns: Don't look at my site :saddowns:
So Confluence has decided that they would like to get rid of their customers by pricing everyone into the cloud. Got to listen in on a FAQ with their local distributor and it was just all their customers hammering them about EU privacy laws and how the datacenter license is gonna cost several magnitudes more than what everyone is paying now (and therefore not viable).
It is clear that Atlassian thought they were very smart to just remove on-premise installs and force everyone into their cloud, but somehow they missed the fact that it would be ILLEGAL for EU companies to use their cloud as it does not have any guarantees against hosting in the US.
They didn't even pull a Microsoft by making Cloud Licensing so much cheaper that you really have to consider it, they just made on-premise licensing so outrageously expensive that no one can afford it.

tl;dr: We and every other company in my country is replacing confluence.

orange sky
May 7, 2007

Hey guys, I'm currently testing the remote execution of some powershell scripts and I've hit a snag with the mcafee/security team - when I'm running the script I have the parameter -ExecutionPolicy Unrestricted, and it's being blocked by McAfee by a rule that forbids that parameter. Understandable.

Usually in this case I will let the team know what process is starting this action, so they can add it to an allow list, but the team is saying that they can't do that - McAfee checks for the powershell parameter before looking for the process that started it, so there's no way to add the executable to an allow list.

I'm not aware of the inner workings of McAfee, but this is weird to me because in previous similar situations there wasn't a problem at all with adding an exe to an exclusion - can anyone confirm if their arguments are correct?

SEKCobra
Feb 28, 2011

Hi
:saddowns: Don't look at my site :saddowns:

orange sky posted:

Hey guys, I'm currently testing the remote execution of some powershell scripts and I've hit a snag with the mcafee/security team - when I'm running the script I have the parameter -ExecutionPolicy Unrestricted, and it's being blocked by McAfee by a rule that forbids that parameter. Understandable.

Usually in this case I will let the team know what process is starting this action, so they can add it to an allow list, but the team is saying that they can't do that - McAfee checks for the powershell parameter before looking for the process that started it, so there's no way to add the executable to an allow list.

I'm not aware of the inner workings of McAfee, but this is weird to me because in previous similar situations there wasn't a problem at all with adding an exe to an exclusion - can anyone confirm if their arguments are correct?

Just sign your loving script if you are using it in production.

orange sky
May 7, 2007

SEKCobra posted:

Just sign your loving script if you are using it in production.

Disregard this, I'm a dumbass :) (all the scripts are signed, yes, btw)

orange sky fucked around with this message at 11:18 on Oct 30, 2020

mattfl
Aug 27, 2004

Anyone else in healthcare get emails about the FBI saying there is a large cyber attack on hospital systems going on right now?


*company name* has been notified by the Department of Health and Human Services (HHS) and the FBI about a potential ransomware threat targeting healthcare and public health sectors through email and embedded links and/or attachments. While this is a significant risk, we want to reassure you that we are prepared in the event we are targeted.

As our information security teams have evaluated our response to this threat, it has been determined that the use of personal email services on our networks and devices pose a significant risk. These email services include: Outlook.com, Gmail, Yahoo Mail, Hotmail and others. Therefore, effectively immediately, access to personal email services from *company name* laptops and desktops has been disabled regardless of what network they are connected to. Access to these email services through the *company name* network has also been disabled.


There's been some major hospital hacks in the past couple months and it sounds like they are starting to take security seriously finally or at least pretending too. Giving doctors whatever access they want might finally come to and end.....lol no it won't.

SEKCobra
Feb 28, 2011

Hi
:saddowns: Don't look at my site :saddowns:

mattfl posted:

Anyone else in healthcare get emails about the FBI saying there is a large cyber attack on hospital systems going on right now?


*company name* has been notified by the Department of Health and Human Services (HHS) and the FBI about a potential ransomware threat targeting healthcare and public health sectors through email and embedded links and/or attachments. While this is a significant risk, we want to reassure you that we are prepared in the event we are targeted.

As our information security teams have evaluated our response to this threat, it has been determined that the use of personal email services on our networks and devices pose a significant risk. These email services include: Outlook.com, Gmail, Yahoo Mail, Hotmail and others. Therefore, effectively immediately, access to personal email services from *company name* laptops and desktops has been disabled regardless of what network they are connected to. Access to these email services through the *company name* network has also been disabled.


There's been some major hospital hacks in the past couple months and it sounds like they are starting to take security seriously finally or at least pretending too. Giving doctors whatever access they want might finally come to and end.....lol no it won't.

Don't worry, you can still execute viruses on your PC, because we are only disabling access to your private mailbox so you can't receive links to executables, we aren't actually removing your software run permissions, that would be ridiculous :v:

mattfl
Aug 27, 2004

SEKCobra posted:

Don't worry, you can still execute viruses on your PC, because we are only disabling access to your private mailbox so you can't receive links to executables, we aren't actually removing your software run permissions, that would be ridiculous :v:

God this is so true, if a doc wants full admin rights on his PC, he gets full admin rights on his PC.

uhhhhahhhhohahhh
Oct 9, 2012

mattfl posted:

Anyone else in healthcare get emails about the FBI saying there is a large cyber attack on hospital systems going on right now?


*company name* has been notified by the Department of Health and Human Services (HHS) and the FBI about a potential ransomware threat targeting healthcare and public health sectors through email and embedded links and/or attachments. While this is a significant risk, we want to reassure you that we are prepared in the event we are targeted.

As our information security teams have evaluated our response to this threat, it has been determined that the use of personal email services on our networks and devices pose a significant risk. These email services include: Outlook.com, Gmail, Yahoo Mail, Hotmail and others. Therefore, effectively immediately, access to personal email services from *company name* laptops and desktops has been disabled regardless of what network they are connected to. Access to these email services through the *company name* network has also been disabled.


There's been some major hospital hacks in the past couple months and it sounds like they are starting to take security seriously finally or at least pretending too. Giving doctors whatever access they want might finally come to and end.....lol no it won't.

Yeah, it's about Ryuk specifically. I'm in the UK, don't think the NHS has had a major hit yet but we've been locking stuff down temporarily and acting as if we've basically already got it.

BaseballPCHiker
Jan 16, 2006

Yeah a friend works at a hospital that has already been hit. Apparently they've been down for 3 days now. He actually has to come into the office to get a paper check since they couldnt even do direct deposit.

From what he has said their IT has always been poo poo so he wasnt that surprised. I'll be real interested to hear if this place just pays the ransom or how they rebuild.

Bob Morales
Aug 18, 2006


Just wear the fucking mask, Bob

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

mattfl posted:

Anyone else in healthcare get emails about the FBI saying there is a large cyber attack on hospital systems going on right now?





Everyone is going nuts around here about it

mattfl
Aug 27, 2004

Bob Morales posted:



Everyone is going nuts around here about it

We got a similar scary sound official email yesterday too with a PDF attached that talks about Ryuk and then this morning the blocking all outside email notice.


UNCLASSIFIED // FOR OFFICIAL USE ONLY – PUBLIC AND MEDIA RELEASE IS NOT AUTHORIZED

(U) ATTN: Region 5 LE; ILOs – All; Governance Board, Steering Committee; CFIX Cyber Working Group; Region 5 IT Managers; Region 5 School Safety Specialists; Region 5 Private Sector Working Group; Federal LE; CFIX Staff:

(U) CFIX Alert #: A-20-1029-33

(U) Message Type: Cyber Situational Awareness

(U) Incident Type: CISA Joint Alert Targeting Healthcare Sector Includes Public DNS Resolver as IOC

(U) Date & Time: 28 October 2020, 2100 EST
(U) Incident Description: The Cybersecurity and Infrastructure Security Agency (CISA), the Federal Bureau of Investigation (FBI), and the U.S. Department of Health and Human Services (HHS) have released Joint Cybersecurity Alert #AA20-302A “Ransomware Activity Targeting the Healthcare and Public Health Sector” – which includes information regarding an increased cybersecurity threat targeting U.S. healthcare facilities and public health departments.
(U//FOUO) One of the Indicators of Compromise for the Anchor_DNS malware listed in the alert, IP address 193[.]183[.]98[.]66, has been historically identified as a public DNS resolver. It is possible that this IOC is a false positive, yet if it is accurate and this public DNS server is compromised, it could lead to compromise of other websites that are utilizing this public resolver to translate the web address to the corresponding IP. Additionally, if this is a false positive and recipients of this alert indiscriminately add all IOCs to block lists, it could affect DNS resolution for those organizations and potentially disrupt internet service.

(U) Source(s): (U) CISA, https://us-cert.cisa.gov/ncas/alerts/aa20-302a; (U//FOUO) CFIX, based on information from CFIX partners, 29 October 2020

(U) Reporting Notice: The Central Florida Intelligence Exchange (CFIX) is sharing this reporting for your information. When the CFIX obtains reliable information that enhances or adds value to what is being reported the CFIX will send to partners as appropriate. The CFIX continuously monitors for a Central Florida nexus to all cyber incidents and will provide significant updates as necessary; the CFIX also asks that any additional information obtained via non-open source reporting be coordinated through the CFIX Cyber email: CFIX-Cyber@ocfl.net

Respectfully,

Central Florida Intelligence Exchange (CFIX) - 212
Hours of Operation: 0800 – 1700 (M-F)
Main Line: 407-858-3950
CFIX After Hours # (On-Call Analyst): 407-509-4084
E-Mail: CFIX-Cyber@ocfl.net

Bob Morales
Aug 18, 2006


Just wear the fucking mask, Bob

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

We've been blocking personal email as long as I can remember.

I want the job at the state government office that sends out scary emails :allears:

mattfl
Aug 27, 2004

Bob Morales posted:

We've been blocking personal email as long as I can remember.

I want the job at the state government office that sends out scary emails :allears:

I can't wait for a doctor or someone who thinks they are more important than they actually are to email/put in a ticket stating they need to access personal email that they can't do their job without it and I get to shut them the gently caress down lol

I have a list of about 5 people who I think once they get in this morning will escalate directly to the hospital COO about this and complain lol

Bob Morales
Aug 18, 2006


Just wear the fucking mask, Bob

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

"Get your own tablet/phone/laptop, and join the guest wireless network."

The only thing we filter on that is porn.

stevewm
May 10, 2005

SEKCobra posted:

So Confluence has decided that they would like to get rid of their customers by pricing everyone into the cloud.


Our POS/ERP/EDI software company has made it well known they would really really like all customers to be on their "cloud" option and get rid of on-premise installs entirely.

Except their cloud option is 6x the price. And they are in the middle of moving the entire thing into Azure and sent out a letter saying the price is going to increase another 15-20% over the next year because of this move.

My company and several of their largest customers (who are colleagues) are still on-premise. We have all let them know in the event they attempt to force us into their "cloud" they will lose our business and the 6 figures a year that we already pay them.


Their "cloud" option is literally just a "lifted and shifted" copy of a on-premise system.

Sickening
Jul 16, 2007

Black summer was the best summer.

stevewm posted:

Our POS/ERP/EDI software company has made it well known they would really really like all customers to be on their "cloud" option and get rid of on-premise installs entirely.

Except their cloud option is 6x the price. And they are in the middle of moving the entire thing into Azure and sent out a letter saying the price is going to increase another 15-20% over the next year because of this move.

My company and several of their largest customers (who are colleagues) are still on-premise. We have all let them know in the event they attempt to force us into their "cloud" they will lose our business and the 6 figures a year that we already pay them.


Their "cloud" option is literally just a "lifted and shifted" copy of a on-premise system.

I assume the cloud infrastructure is a million times easier to support than their customers on prem solutions.

stevewm
May 10, 2005

Sickening posted:

I assume the cloud infrastructure is a million times easier to support than their customers on prem solutions.

Possibly... though the software isn't really complex at all. It's a straight .NET app running over RemoteApp with a MS SQL database. The "cloud" version is exactly this as well. They have you setup a IPSec VPN to their data center and then give you a RDP file to run to access it. 99.9% of our support tickets are either verified bugs or us just not knowing how to do things. Its extremely rare for there to be a problem traced back to being on-premise.

I feel their cloud ambitions are more revenue driven than anything. We already pay near 6 figures in ongoing support a year. The cloud option has you continue to pay this plus the additional cloud fees. For our size the cloud fees amount to 6 figures. It makes sense for a smaller company... but at our size and the amount of money they want, we could upgrade/replace our entire server infrastructure that runs this software every other year! It just doesn't make fiscal sense to go with their cloud option.

skipdogg
Nov 29, 2004
Resident SRT-4 Expert

Yeah the Atlassian announcement left a really bad taste in our mouth. Our engineering departments are so deep into their software though I don't think we have a choice but to pay for datacenter licenses. The extra 3 bucks a month for SSO is BS as well.

Sickening
Jul 16, 2007

Black summer was the best summer.

stevewm posted:

Possibly... though the software isn't really complex at all. It's a straight .NET app running over RemoteApp with a MS SQL database. The "cloud" version is exactly this as well. They have you setup a IPSec VPN to their data center and then give you a RDP file to run to access it. 99.9% of our support tickets are either verified bugs or us just not knowing how to do things. Its extremely rare for there to be a problem traced back to being on-premise.

I feel their cloud ambitions are more revenue driven than anything. We already pay near 6 figures in ongoing support a year. The cloud option has you continue to pay this plus the additional cloud fees. For our size the cloud fees amount to 6 figures. It makes sense for a smaller company... but at our size and the amount of money they want, we could upgrade/replace our entire server infrastructure that runs this software every other year! It just doesn't make fiscal sense to go with their cloud option.

Its more than that though. They have to factor in your issues locally to your datacenter/environment in their support tickets vs more streamlined support in azure. If anything its probably both.

skipdogg
Nov 29, 2004
Resident SRT-4 Expert

So I'm in the interview process for a job right now and I'm conflicted about taking the gig. Just finished the 2nd interview, have a good feeling about it, expecting to hear back early next week.

New company would be very solid, benefits are very good, more vacation days, generous tuition/training allowance, WFH, etc. Company is constantly in the top 100 places to work in the country list, this is the kind of place people retire from. I'd be making a little more base, but the 401k match and bonus would probably add 20% to my total comp right now.

I'd be taking a step back though. I'm currently in an 100% project/ architecture role right now. No operations at all. I'm involved with pretty much all of IT infrastructure, but focus on Windows/Active Directory. New job would be slightly more base money, and way more stable.... but I'd be 100% focused on Active Directory, a mix of project and ticket request work, and back in a once every 4 week on-call rotation. I did the on call thing when I was younger... it sucked, but I was supporting a 24/7 call center. This would be way less calls maybe 1 a week. Unclear if there is a premium for being on call. This would also be a trade off for getting my foot in the door, internal mobility is pretty good. Might be a pay your dues situation before you can move to other areas.

I'm not unhappy at my current job, but I have major reservations about the outlook of the company. Our long time CEO was recently fired, and private equity folks appointed a new CEO who's heavily incentivized to make the stock price go up. He has a history of chopping up companies and selling them piecemeal. I've been through this playbook before. Costs and benefits are going to be cut, and honestly I don't see my current team surviving the inevitable downsizing that will probably come next year. Who needs to architect new solutions when you're not spending any money?

Is it worth it to take a step down in job roles(for more money), to get my foot in the door at a better work environment with much better long term upward mobility prospects? Am I just being a big baby scared of change? (yes I am)



jaegerx, you're local to me and can probably figure out where the potential job is at... if you have any opinion or feedback about the place I'd appreciate it.

skipdogg fucked around with this message at 18:34 on Oct 30, 2020

The Fool
Oct 16, 2003


skipdogg posted:

Is it worth it to take a step down in job roles(for more money),
No.

quote:

to get my foot in the door at a better work environment with much better long term upward mobility prospects?
Yes.

quote:

Am I just being a big baby scared of change? (yes I am)
Yes.

2/3, you should probably take it.

in a well actually
Jan 26, 2011

dude, you gotta end it on the rhyme

skipdogg posted:

So I'm in the interview process for a job right now and I'm conflicted about taking the gig. Just finished the 2nd interview, have a good feeling about it, expecting to hear back early next week.

New company would be very solid, benefits are very good, more vacation days, generous tuition/training allowance, WFH, etc. Company is constantly in the top 100 places to work in the country list, this is the kind of place people retire from. I'd be making a little more base, but the 401k match and bonus would probably add 20% to my total comp right now.

I'd be taking a step back though. I'm currently in an 100% project/ architecture role right now. No operations at all. I'm involved with pretty much all of IT infrastructure, but focus on Windows/Active Directory. New job would be slightly more base money, and way more stable.... but I'd be 100% focused on Active Directory, a mix of project and ticket request work, and back in a once every 4 week on-call rotation. I did the on call thing when I was younger... it sucked, but I was supporting a 24/7 call center. This would be way less calls maybe 1 a week. Unclear if there is a premium for being on call. This would also be a trade off for getting my foot in the door, internal mobility is pretty good. Might be a pay your dues situation before you can move to other areas.

I'm not unhappy at my current job, but I have major reservations about the outlook of the company. Our long time CEO was recently fired, and private equity folks appointed a new CEO who's heavily incentivized to make the stock price go up. He has a history of chopping up companies and selling them piecemeal. I've been through this playbook before. Costs and benefits are going to be cut, and honestly I don't see my current team surviving the inevitable downsizing that will probably come next year. Who needs to architect new solutions when you're not spending any money?

Is it worth it to take a step down in job roles(for more money), to get my foot in the door at a better work environment with much better long term upward mobility prospects? Am I just being a big baby scared of change? (yes I am)



jaegerx, you're local to me and can probably figure out where the potential job is at... if you have any opinion or feedback about the place I'd appreciate it.

All you had to say was private equity. Get the gently caress out.

Podima
Nov 4, 2009

by Fluffdaddy
Yes that seems extremely cut and dry. Take the new opportunity before your current job vanishes.

Wibla
Feb 16, 2011

skipdogg posted:

I'm not unhappy at my current job, but I have major reservations about the outlook of the company. Our long time CEO was recently fired, and private equity folks appointed a new CEO who's heavily incentivized to make the stock price go up. He has a history of chopping up companies and selling them piecemeal. I've been through this playbook before. Costs and benefits are going to be cut, and honestly I don't see my current team surviving the inevitable downsizing that will probably come next year. Who needs to architect new solutions when you're not spending any money?

Run the gently caress away from this place.

Hollow Talk
Feb 2, 2014

skipdogg posted:

Yeah the Atlassian announcement left a really bad taste in our mouth. Our engineering departments are so deep into their software though I don't think we have a choice but to pay for datacenter licenses. The extra 3 bucks a month for SSO is BS as well.

That SSO thing really annoys me as well. I suppose somebody realised they didn't charge extra for the G Suite sync, and by God can't we have anything in the Atlassian world that doesn't nickel-and-dime you every single step of the way.

Obligatory: https://sso.tax/

The Fool
Oct 16, 2003


Hollow Talk posted:

That SSO thing really annoys me as well. I suppose somebody realised they didn't charge extra for the G Suite sync, and by God can't we have anything in the Atlassian world that doesn't nickel-and-dime you every single step of the way.

Obligatory: https://sso.tax/

minor brag: I've contributed to this project

skipdogg
Nov 29, 2004
Resident SRT-4 Expert

The Fool posted:



2/3, you should probably take it.

Yeah, this is what 2 of my IT worker friends and my wife are telling me to do.

If the offer comes in reasonable, I'll take it.

Wonder if I can pull a Sickening and work both jobs for a while...


edit: Holy Crap, They already made me an offer... they bumped the title up to Senior and it's a 23K base pay bump.

skipdogg fucked around with this message at 19:42 on Oct 30, 2020

Internet Explorer
Jun 1, 2005





skipdogg posted:

Yeah, this is what 2 of my IT worker friends and my wife are telling me to do.

If the offer comes in reasonable, I'll take it.

Wonder if I can pull a Sickening and work both jobs for a while...


edit: Holy Crap, They already made me an offer... they bumped the title up to Senior and it's a 23K base pay bump.

Congrats! If I remember correctly you've been at the current place for a while. With all the other poo poo going on, you should seriously consider it! Sometimes change just for change's sake is nice.

Podima
Nov 4, 2009

by Fluffdaddy

skipdogg posted:

Yeah, this is what 2 of my IT worker friends and my wife are telling me to do.

If the offer comes in reasonable, I'll take it.

Wonder if I can pull a Sickening and work both jobs for a while...


edit: Holy Crap, They already made me an offer... they bumped the title up to Senior and it's a 23K base pay bump.

:yotj:

Sickening
Jul 16, 2007

Black summer was the best summer.

skipdogg posted:

Yeah, this is what 2 of my IT worker friends and my wife are telling me to do.

If the offer comes in reasonable, I'll take it.

Wonder if I can pull a Sickening and work both jobs for a while...


edit: Holy Crap, They already made me an offer... they bumped the title up to Senior and it's a 23K base pay bump.

You should pull a sickening my dude, you are worth it. If the old job starts being a pain, quit.

I do have :yotj: with another moonlighting gig to replace the one I was just laid off of. Technically I have just accepted the offer, I now have 3 weeks of background checks and other HR poo poo before I even get a start date. Its for one of the big evil corps, but they are 100% work from home. I already talked to my new Director, my job is heavily silo'd and despite him trying to hype up its work load, it seems awful light which is perfect.

I was also able to land this job without even listing my current job in my resume. Technically there is showing a 4 month gap, but the moonlighting job basically makes up the rest of the year. Now I can rearrange which jobs I list as needed and their background checks will still line up. I am pretty pumped as this should make continued moonlighting easier going forward.

I am both happy and sad about this as my wife has finally decided to resign from her teaching job because her school isn't handling covid particularly well. My goals of eventually meeting biden's new tax bracket are probably squashed forever.

For those brave souls out there, I would get into it was the getting is good because the more companies that go remote this entire thing is going to get squashed eventually. HR departments and 3rd party services are going to probably target people trying to do this with tools that better track job history by sharing more and more data with each other. For now, I am going to get mine though.

Adbot
ADBOT LOVES YOU

skipdogg
Nov 29, 2004
Resident SRT-4 Expert

Internet Explorer posted:

Congrats! If I remember correctly you've been at the current place for a while. With all the other poo poo going on, you should seriously consider it! Sometimes change just for change's sake is nice.

Today is actually my 17th anniversary if you count the 9 months I was a contractor. Been through multiple acquisitions and job roles though. Haven't changed companies on my own since I was 22.....

The considering part is over. I'm taking the job. +23K base, plus more vacation, better health insurance, better benefits overall, plus an actual bonus that generally pays out. If they want to pay me 115K plus bonus a year to create GPO's and monitor domain controllers fine with me. I don't mind being on call every 4 weeks either at that pay.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply