|
I'm struggling with something on provisioning GKE resources using helm. For typical workloads, where you have one Deployment with one Service and one ServiceAccount, I've got it well figured out. I think. At least things are working well. But now I'm trying to deploy Jobs using helm. The pattern is multiple jobs launched with different parameters (the alternative was setting up a queue and using parallel mode but this was easier). This means many Jobs, one ServiceAccount. I set up the charts similarly to the Deployment charts. When I go to helm install these jobs, the first one works fine, but then all the subsequent ones fail because the ServiceAccount already exists. I should add that the ServiceAccount gets his permissions using the iam-policy-binding between GCP service accounts and k8s service accounts. The first thing is, I guess there is no way to say, it's OK if it exists? If not, I see several possibilities. We could use a unique service account for each Job, which would mean that we would have to also run the iam-policy-binding for each Job deploy. Or, we could separate the ServiceAccount from the chart, and when deploying a chunk of jobs, first use kubectl to directly create the service account, then install the serviceaccount-less chart for each job. Or I guess, combine the Jobs into one Job by setting up a work queue. Am I missing something easier? Any suggestions here?
|
#
?
Aug 13, 2020 11:07
|
|
|
|
| # ? Apr 19, 2024 05:01 |
|
|
This is a helm is stupid problem. if you rendered out the yaml kubernetes would happily let you do applies on top of each other, as it’s designed as a convergent API. Helm is very dumb and tries to make everything atomic. Remove the Service Account from the helm chart or rewrite it so that there is one master service account and all the jobs reference it. If that’s not possible then you’ll probably need to separate it out as it’s own unique chart.
|
|
#
?
Aug 13, 2020 12:51
|
|
|
Thanks, makes sense. Yeah, that seems to be the easiest option.
|
|
#
?
Aug 13, 2020 14:27
|
|
|
I can think of like 3 paid products that do this off the top of my head Is there a free script that will scan your AWS infrastructure, check to make sure any SG that aren't white listed, don't have 5432, 22, 80 etc open to 0.0.0.0/0 and then emails out a report This seems like 20 lines of Python but I don't feel like writing it
|
|
#
?
Aug 14, 2020 21:38
|
|
|
Hadlock posted:I can think of like 3 paid products that do this off the top of my head Yeah you could do it with 20 lines of python, but I think that's only the tip of the iceberg about what can go wrong with AWS security configs What about s3 buckets open to the world, AWS keys that haven't been rotated, etc? At my company we used RedLock to monitor this stuff. Even creates Jira tickets automatically when it finds poo poo
|
|
#
?
Aug 15, 2020 01:13
|
|
|
We've used ScoutSuite in the past with reasonable success: https://github.com/nccgroup/ScoutSuite. But now we're starting a project to roll out Config to all of our accounts and set Organisation-level compliance rules.
|
|
#
?
Aug 15, 2020 01:46
|
|
|
Some of that stuff is free via trusted advisor isn’t it? https://aws.amazon.com/premiumsupport/technology/trusted-advisor/best-practice-checklist/ I don’t know what the costs are if you did everything, but if it saves you time it might be worth it.
|
|
#
?
Aug 15, 2020 02:08
|
|
|
I think AWS Inspector has a "network reachability" report of some kind that will report anything accessible from the public internet. https://docs.aws.amazon.com/inspector/latest/userguide/inspector_network-reachability.html I didn't touch this one myself, so I dunno if it's easy to get that info and just grep, but AWS does tout how it's based on cool automated reasoning stuff to ensure everything it reports is real.
|
|
#
?
Aug 15, 2020 04:03
|
|
|
whats for dinner posted:We've used ScoutSuite in the past with reasonable success: https://github.com/nccgroup/ScoutSuite. But now we're starting a project to roll out Config to all of our accounts and set Organisation-level compliance rules. This is what we do with our relatively-large Org w/ a bunch of child accounts. All the Config results go through an aggregator to our Infosec team's account so they get a dashboard of everyone else's resource compliance. (We're probably going to have to wire the config SNS topic up to something too, so we can get better active notifications in external systems.) We did end up writing a bunch of "Custom Rules" w/ a forked version of rdklib since the AWS managed rules don't do exemptions/allowlists and we have a bunch of corner cases and stuff for networking rules. All the custom rules are managed in TF so it's pretty simple to add new ones / promote changes through our different env's root accounts. e: also we interact with Control Tower which creates a hosed-up AWS Config setup for our use case, which is another reason we forked rdklib JehovahsWetness fucked around with this message at 15:50 on Aug 15, 2020 |
|
#
?
Aug 15, 2020 15:31
|
|
|
Our aws misconfiguration situation is wild. We sell a product that does it well, recently acquired a company that does a great job at it, and I wrote a tool that does it decently before the first 2 things existed. Please don't dox me, ok?
|
|
#
?
Aug 15, 2020 15:42
|
|
|
JehovahsWetness posted:This is what we do with our relatively-large Org w/ a bunch of child accounts. All the Config results go through an aggregator to our Infosec team's account so they get a dashboard of everyone else's resource compliance. (We're probably going to have to wire the config SNS topic up to something too, so we can get better active notifications in external systems.) Your Config setup sounds basically like what we're heading toward. A lot of it's still in the experimentation phase for us, so I'm still trying to figure out how best to roll things out. We use Terraform for infrastructure management but getting a new child account ready in our org is still a really manual process (creating IAM roles, etc.) that I think we can get Control Tower to look after but it sounds like from your experience maybe not? I know our security analyst would be thrilled if as part of creating a new account with Control Tower it also turned on Config and immediately started forwarding to the aggregator in the security account.
|
|
#
?
Aug 16, 2020 03:29
|
|
|
Yeah I also worked for a config auditing company in a previous life I will check out scout suite
|
|
#
?
Aug 16, 2020 05:21
|
|
|
Any recommendations for tutorials/courses on Ansible? I'm applying for cloud engineer / DevOps jobs and a lot of them want IaC knowledge. I've done the HashiCorp Terraform tutorials but it seems like frequently Ansible and Terraform are used in conjunction.
|
|
#
?
Aug 25, 2020 01:39
|
|
|
22 Eargesplitten posted:Any recommendations for tutorials/courses on Ansible? I'm applying for cloud engineer / DevOps jobs and a lot of them want IaC knowledge. I've done the HashiCorp Terraform tutorials but it seems like frequently Ansible and Terraform are used in conjunction. https://leanpub.com/ansible-for-devops He also did a full YouTube series on it. Geerling is pretty deep in the Ansible community and a great starting point for the basics.
|
|
#
?
Aug 25, 2020 01:41
|
|
|
Sweet, thanks. I'll take a look at it when the queue slows down later tonight.
|
|
#
?
Aug 25, 2020 02:23
|
|
|
https://twitter.com/emilyst/status/1290533146509733888
|
|
#
?
Aug 25, 2020 04:10
|
|
|
Why the hell is maintaining task definitions for ECS such a pain in the rear end? Jfc
|
|
#
?
Aug 25, 2020 05:32
|
|
|
Beamed posted:Why the hell is maintaining task definitions for ECS such a pain in the rear end? Jfc Lmao. Cause ECS sucks. And when ppl say it’s good or that they should use it instead of K8s I roll on the floor laughing.
|
|
#
?
Aug 25, 2020 13:15
|
|
|
You have to use CloudFormation with ECS, and it's a great service as long as you do. This is the same for everything else in AWS, as well. Given my experience with k8s I have to imagine you would either need to live in a fantasy world or to have never worked with a good toolchain to consider it inferior to kubectl apply.
|
|
#
?
Aug 25, 2020 17:08
|
|
|
Beamed posted:Why the hell is maintaining task definitions for ECS such a pain in the rear end? Jfc You should really be making your stacks into something like cloudformation, cdk, terraform, ansible or whatever and just use that to manage it. You really shouldn't need to update it too much its set up right. They also have the older cli commands or this new thing: https://github.com/aws/copilot-cli
|
|
#
?
Aug 25, 2020 19:01
|
|
|
Favorite SQL editor/ide thing? For Mac, preferably with good postgres support. I really liked pgadmin 3 as a separate app. I've been using pgadmin 4 but I really don't like having my SQL editor tied to the stability of my browser, especially with all the bullshit plugins IT auto installs on my laptop. Considering going back to pgadmin 3, but looking for other options
|
|
#
?
Aug 27, 2020 20:26
|
|
|
VS Code?
|
|
#
?
Aug 27, 2020 20:27
|
|
|
Hadlock posted:Favorite SQL editor/ide thing? For Mac, preferably with good postgres support. I really liked pgadmin 3 as a separate app. I've been using pgadmin 4 but I really don't like having my SQL editor tied to the stability of my browser, especially with all the bullshit plugins IT auto installs on my laptop. Considering going back to pgadmin 3, but looking for other options Paid: jet brains, DataGrip is the database specific one Free: VScode
|
|
#
?
Aug 27, 2020 20:34
|
|
|
DBeaver keeps getting better and better. If you haven't used it in a year or more, give it a try - they've been dropping releases every two weeks like clockwork and almost always with significant new features or improvements. Nowadays I even use it with SQL Server instead of SSMS, and if that's not high praise I don't know what is.
|
|
#
?
Aug 27, 2020 20:41
|
|
|
+1 for dbeaver
|
|
#
?
Aug 28, 2020 02:23
|
|
|
Datagrip or intellij with the DB plugin
|
|
#
?
Aug 28, 2020 15:41
|
|
|
JHVH-1 posted:You really shouldn't need to update it too much its set up right. I manage mine with TF but everytime my app needs new env variables, updated ones, etc., I need to push an infra change. And that's always at least a 25 minute process of adding the env var to the infra and rerunning the code deployment.
|
|
#
?
Sep 9, 2020 02:43
|
|
|
What do you goons' k8s clusters look alike after you provision a new cluster? We're eyeing dex for identity but curious if anyone else is using a similar setup with more complex addons
|
|
#
?
Sep 11, 2020 23:17
|
|
|
Gyshall posted:What do you goons' k8s clusters look alike after you provision a new cluster? We're eyeing dex for identity but curious if anyone else is using a similar setup with more complex addons prometheus stack, kiam, metrics server/keda, cluster autoscaler, some cronjob/job cleaner, spot instance termination handler, externalDNS, ingress nginx, our own internal implementation of a cloud controller to handle some weird federation stuff we do. We'll soon have a mutating admission controller to work around a serious problem we have with cardinality breaking our graphite timeseries stack. basically having a pool of reusable identifiers rather than constantly making new timeseries that are ID'd by ephemeral information like podIP-service name pairs. I think those are the main components. Registration to spinnaker and internal consul/service discovery stuff is there too but that's not really deployed in-cluster. We use dex for identity too, we use it to mint OIDC tokens that contain user ldap groups that we have rbac targetting. Methanar fucked around with this message at 23:31 on Sep 11, 2020 |
|
#
?
Sep 11, 2020 23:23
|
|
|
If you have an existing oidc provider; aka auth0 or Okta, you can(and imo, should) skip dex and directly auth against it. The oidc login plugin in krew is sweet and works with a ton of providers. My home cluster uses my gsuite org for example. Edit; which to say dex is fine, but I like to forgo it if I don’t need it AWS auth is also nifty if you run EKS or kops.
|
|
#
?
Sep 14, 2020 00:53
|
|
|
freeasinbeer posted:If you have an existing oidc provider; aka auth0 or Okta, you can(and imo, should) skip dex and directly auth against it. The oidc login plugin in krew is sweet and works with a ton of providers. Okta wants to charge you like 3 extra dollars per user, per month to include ldap groups in its tokens.
|
|
#
?
Sep 14, 2020 01:17
|
|
|
Use okta to federate AWS account access and then use AWS auth for wks Ez pz
|
|
#
?
Sep 14, 2020 01:58
|
|
|
Beamed posted:I manage mine with TF but everytime my app needs new env variables, updated ones, etc., I need to push an infra change. And that's always at least a 25 minute process of adding the env var to the infra and rerunning the code deployment. With ECS at least you can now point environment variables to a secrets manager, and you just have to update the secret there and most likely refresh the service afterwards. It avoids having to continually update the task definition at least. I don’t know if you change it on the fly if something like nodejs or whatever polling the environment again would pick it up on a running container. (Now I’m curious to test that sometime) Otherwise how long that takes probably depends on how long your app spins up and how long before your health checks go healthy. New container needs to spin up, take traffic, old container has connects drain and then removed before everything is complete and stable again.
|
|
#
?
Sep 14, 2020 03:01
|
|
|
Methanar posted:Okta wants to charge you like 3 extra dollars per user, per month to include ldap groups in its tokens. Doesn’t have to be ldap groups, although to your point that is dependent on specifics of your existing setup and desired setup. I know we just use groups defined in Okta itself.
|
|
#
?
Sep 14, 2020 03:07
|
|
|
I've spent the last 6 hours reading about bpf and ebpf which is completely different and kernel networking poo poo and cilium's code and netfilter and xdp and cloudflare and cilium blogs and I understand nothing. I'm pretty sure I found a race condition in cilium and now I'm stuck in an endless blackhole of reading about low level kernel networking facilities.
|
|
#
?
Sep 29, 2020 06:19
|
|
|
My employer has decided to go all in on SAFe and I have spent 6 days in the last 3 weeks in interminable training. With 3 days of PI planning happening next week that is all coming from an executive-groomed backlog. Furiously applying for jobs elsewhere.
|
|
#
?
Oct 2, 2020 02:48
|
|
|
Get them to certify you first if you can. That stuff is a hot commodity these days.
|
|
#
?
Oct 2, 2020 04:56
|
|
|
Warbird posted:Get them to certify you first if you can. That stuff is a hot commodity these days. Wow, really? This stuff just seems so... frustrating. My team currently services so many other teams (bunch of dev teams, professional services, helpdesk) as well as being responsible for all the platform infrastructure that the concept of trying to plan 5 2-week sprints of work seems like wishful thinking.
|
|
#
?
Oct 2, 2020 05:19
|
|
|
Since when has demand ever been for useful things? 
|
|
#
?
Oct 2, 2020 05:36
|
|
|
|
| # ? Apr 19, 2024 05:01 |
|
|
Warbird posted:Since when has demand ever been for useful things? gently caress... ya got me 
|
|
#
?
Oct 2, 2020 05:52
|
|