|
the talent deficit posted:don't create access keys in cfn. grant users the ability to create and manage their own keys if they need them or use federation to allow them to assume roles instead Yeah I’d rather not create them in cloud formation but how can I federate Kubernetes pods? I guess I can google it.
|
#
?
Apr 17, 2018 02:07
|
|
|
|
| # ? Apr 25, 2024 18:21 |
|
|
AWWNAW posted:Yeah I’d rather not create them in cloud formation but how can I federate Kubernetes pods? I guess I can google it. IAM+sts? Or vault?
|
|
#
?
Apr 17, 2018 02:09
|
|
|
freeasinbeer posted:IAM+sts? I think Vault is being evaluated but not live yet at work. And I don’t want to jump through a bunch of hoops to do role assumption poo poo via cluster nodes.
|
|
#
?
Apr 17, 2018 02:13
|
|
|
AWWNAW posted:I think Vault is being evaluated but not live yet at work. And I don’t want to jump through a bunch of hoops to do role assumption poo poo via cluster nodes. I don’t see how sts is functionality different then inserting access keys into k8s secrets. Edit: https://github.com/jtblin/kube2iam Edit2: I was a bit blunt but I’ve been going through the same dilemma and am at a similar impasse. Access keys in secrets aren’t really any securer then sts and assuming roles. freeasinbeer fucked around with this message at 02:31 on Apr 17, 2018 |
|
#
?
Apr 17, 2018 02:19
|
|
|
What is the best way to run jobs/executable on demand in AWS? ECS/Fargate? AWS Batch? Some other mechanism? Our problem: Customer creates project in our web application, which saves project data in the database (Postgresql RDS). Based on the data in the project, we need to train a model. The amount of time it takes to complete the computation depends on the amount of data in the project. It is a linear dependency. Since the computation doesn't need to happen often (only when the data changes, which is relatively rare) we do not want to pay through the nose for a compute-optimized EC2 instance. So, I am looking at ways to do that calculation asynchronously. A docker container that runs my application (C++ based) looks to be the ideal candidate, which is started when data is changed, runs then dies. As an input it needs the project id, then the application goes in the database to fetch the data, it does the calculation, then puts the result (the model) back in the database and potentially (this is optional) notifies someone in some way that the job is complete. Looking through the documentation in AWS is quite confusing which way is the best to go. How can I programatically start the job in AWS? Some notification service? And how do I pass in parameters (said project ID) to the docker image/my executable? Also, where would it be best to store database credentials? In the docker image when it is created? Somehow passed as arguments when the task is created? Some other location? All of this I would know how to solve using RabbitMQ and a pre-made EC2 machine that runs my application, but we don't want to pay for it to run all the time when not needed. Thanks.
|
|
#
?
Apr 17, 2018 19:38
|
|
|
Can you build it in Lambda?
|
|
#
?
Apr 17, 2018 19:41
|
|
|
Lambda has a short maximum execution time and since he mentioned using compute optimized instances I’m guessing that won’t work. Have you looked at spot instances though? Edit - also look at using SNS for your messaging. If lambda will work for you, it’s dead simple to trigger lambda functions off of an SNS topic. deedee megadoodoo fucked around with this message at 19:48 on Apr 17, 2018 |
|
#
?
Apr 17, 2018 19:44
|
|
|
You can trigger based on scheduled events with lambda https://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html Lambda has other triggering options as well there is SNS for messaging, SQS for queuing, S3 can be used as hotfolder. Also didn't know this was a thing till now https://aws.amazon.com/batch/
|
|
#
?
Apr 17, 2018 19:50
|
|
|
very stable genius posted:Lambda has a short maximum execution time and since he mentioned using compute optimized instances I’m guessing that won’t work. Have you looked at spot instances though? Well Lambda can be used to trigger a job, like run something in fargate and then it does its thing and generates what you need and then exits so nothing is left running. Example: https://serverless.com/blog/serverless-application-for-long-running-process-fargate-lambda/
|
|
#
?
Apr 17, 2018 19:52
|
|
|
As far as I can tell for Lambda one can use Javascript, C#, Java kind of languages. Our thing is done in C++ so lmbda is kinda out of the question as far as I can tell. I am studying batch right now, is just that it looks very confusing. Which is why I thought I'd ask.JHVH-1 posted:Well Lambda can be used to trigger a job, like run something in fargate and then it does its thing and generates what you need and then exits so nothing is left running. Oh ... that's an interesting approach.
|
|
#
?
Apr 17, 2018 19:53
|
|
|
AWWNAW posted:Yeah I’d rather not create them in cloud formation but how can I federate Kubernetes pods? I guess I can google it. i thought this was for actual people users, my bad i've only ever used https://github.com/jtblin/kube2iam for iam with k8s pods but i didn't set it up. as far as i know it doesn't require access keys or users at all
|
|
#
?
Apr 17, 2018 20:18
|
|
|
Volguus posted:What is the best way to run jobs/executable on demand in AWS? ECS/Fargate? AWS Batch? Some other mechanism? i solved a similar problem to yours with sqs + cloudwatch alarms + lambda. we'd post new jobs to an sqs queue. an alarm on the queue would fire when the queue was nonempty and that would trigger a lambda that read the sqs message and started the job (in our case an emr cluster).
|
|
#
?
Apr 17, 2018 20:22
|
|
|
Volguus posted:As far as I can tell for Lambda one can use Javascript, C#, Java kind of languages. Our thing is done in C++ so lmbda is kinda out of the question as far as I can tell. You can run anything compiled in lambda if you compile it for the right environment, include the binary in the zip file, and include a wrapper in a supported language that shells out when invoked. E: Doc Hawkins fucked around with this message at 02:54 on Apr 18, 2018 |
|
#
?
Apr 17, 2018 20:25
|
|
|
Doc Hawkins posted:You can run anything compiled in lambda if you compile it for the right environment, include the binary in the zip file, and include a wrapper in a supported language that shells out when invoked. Yes, it takes around 45 seconds to complete right now in a medium 1cpu instance of AWS, 20 seconds to complete on our local workstations with a reasonable (expected) load. So, maybe then aws batch?
|
|
#
?
Apr 17, 2018 21:13
|
|
|
Volguus posted:Yes, it takes around 45 seconds to complete right now in a medium 1cpu instance of AWS, 20 seconds to complete on our local workstations with a reasonable (expected) load. So, maybe then aws batch? Lambda's max timeout is 5 minutes, so it's probably you could get it to work?
|
|
#
?
Apr 18, 2018 02:03
|
|
|
Rapner posted:Lambda's max timeout is 5 minutes, so it's probably you could get it to work? Hmm, potentially yes. It would probably be easier for me to get my stuff compiled on/for a known docker image, then via lambda execute it in FARGATE. I am essentially trying to replicate https://serverless.com/blog/serverless-application-for-long-running-process-fargate-lambda/ . If that won't work, then I'll look for alternatives.
|
|
#
?
Apr 18, 2018 05:15
|
|
|
Acidian posted:That's exactly what it was. I had a feeling it had to do with me being retarded. Not retarded at all, this gets every AWS user from time to time "Why can't I see this thing? Oh... duh..."
|
|
#
?
Apr 19, 2018 03:33
|
|
|
Does anyone know if its possible in cloudformation to do a GetAtt on a resource that's already been created manually? Like, not something from another stack, just an Arn from say an SNS topic you turned on by hand in the console? Yeah I could make a parameter and have the user enter it at runtime, but what's the fun in that. It doesn't look like this is possible but figured I would ask.
|
|
#
?
Apr 19, 2018 20:41
|
|
|
SnatchRabbit posted:Does anyone know if its possible in cloudformation to do a GetAtt on a resource that's already been created manually? Like, not something from another stack, just an Arn from say an SNS topic you turned on by hand in the console? Yeah I could make a parameter and have the user enter it at runtime, but what's the fun in that. It doesn't look like this is possible but figured I would ask. You could add a simple Custom Resource that fires a Lambda function that queries the arn and returns it as an attribute of the custom resource. If you can do it in node or python you can keep the function code in your cfn stack itself. Just drop a Lambda function resource with the code and a custom resource that calls the Lambda function when the stack is created.
|
|
#
?
Apr 20, 2018 08:48
|
|
|
After fooling around with a docker image that holds my application which is launched from a lambda in a Fargate compatible cluster - I've decided to drop it, for now. It just takes too drat long (sometimes even 4 minutes) for the drat image to get launched. Maybe that's the idea i guess, save money on containers but give up time it takes for the thing to execute. So, for now, I went to the old (I believe) auto-scaling option in AWS. Made the launch configuration and the Auto Scaling group, set the conditions to be met to go up and go down and I DOS-ed my application in AWS. It perfectly went up (more instances) and after a while scaled down just like expected so for now I am very happy with this approach. It is probably more expensiv than the other one, but ... oh well. We'll monitor cost closely in the next few weeks. I do have a question though: If I want update my application, it seems that I have to update the image, don't I? And if I update the image it seems that I need a new template, and a new launch configuration from that template and then update the Scaling group to use the new launch configuration, and then to force it to use the new image I have to detach the old instance and tell it to launch new instance. This seems quite ... complicated and awful and long . Is there a better way? Can I automate it? I just need to update certain files in the image and restart 2 systemd services.
|
|
#
?
Apr 21, 2018 20:05
|
|
|
Netflix designed spinnaker to do this, you could also use any of the config management tools and some combo of packer. Any reason why ecs(although I dislike it) doesn’t work?
|
|
#
?
Apr 21, 2018 20:25
|
|
|
Thanks Ants posted:The region thing in AWS being a global setting is fairly annoying - I’d like to see all instances returned and then a column for region that can be filtered. It’s a blast radius thing. When you flip between regions, you are literally flipping to a new instance of AWS that lives somewhere else. As services are launched they start with a single target region and then a new block of service infrastructure is spun up in a new region, and things progress from there. At AWS almost everything is modular. Except for stuff like billing and a few other things. Traditionally we leave it up to the customer to unify things to their liking. But Organizations does a lot to unify stuff across accounts and regions. In your case, where you’d want to see all instances everywhere, we’d expect you to build that report yourself using the API.
|
|
#
?
Apr 21, 2018 20:46
|
|
|
freeasinbeer posted:Netflix designed spinnaker to do this, you could also use any of the config management tools and some combo of packer. Jesus, why does everything have to be so complicated? I'll study spinnaker more, but all I want is to update the image a launch configuration is using. Ideally one command from my side that would launch the build, execute the tests, and if successful deploy. Why ECS doesn't work? Because, as I said before, it takes 5 minutes for the thing to start up and launch the application that is in the container. I manually executed the lambda that started the task. With some notification system (SNS) would potentially take even longer. Everything in AWS seems to just take a long time. Like the other day I created an IAM user to get the AWS ID and secret key for docker deployment, and when I tried to login with it I got internal error for an hour, after which it magically just worked. Maybe stuff needs to propagate to places? No idea. I have absolutely no clue how this AWS monster works at all. No wonder, with such complicated tooling, that there are people whose full-time job is to manage this cloud crap. I haven't tried any other cloud providers so I have no clue if the others are better or worse.
|
|
#
?
Apr 21, 2018 20:55
|
|
|
welcome to devops, the next step is to pick your self medicating poison of choice
|
|
#
?
Apr 21, 2018 21:09
|
|
|
Agrikk posted:It’s a blast radius thing. When you flip between regions, you are literally flipping to a new instance of AWS that lives somewhere else. As services are launched they start with a single target region and then a new block of service infrastructure is spun up in a new region, and things progress from there. Thanks for the explainer
|
|
#
?
Apr 21, 2018 21:14
|
|
|
Volguus posted:Jesus, why does everything have to be so complicated? I'll study spinnaker more, but all I want is to update the image a launch configuration is using. Ideally one command from my side that would launch the build, execute the tests, and if successful deploy. Sorry I forgot you mentioned fargate, I meant the regular version of ecs, if what you were going to be doing is spinning up new nodes with ASGs. Fargate is a special mode of ecs(where it manages the node for you). For what it’s worth spinnaker is super complicated and not worth your time, I was just mentioning it because that is what Netflix built to solve the issue you want in the way you want. I’m spitballing ideas here, but maybe an image that pulls a docker container on startup from ECR and update the docker container. You’d probably also want to update the launch config with an explicit version. There is also beanstalk, it might be easier for a one off. FWIW depending on what you need to packer and baking amis is still decently fast. But yeah you’d need to manage the launch config on the asg.
|
|
#
?
Apr 21, 2018 21:17
|
|
|
I mean this is what heroku is for. But I don’t know a lot about the specifics here
|
|
#
?
Apr 21, 2018 21:19
|
|
|
StabbinHobo posted:welcome to devops, the next step is to pick your self medicating poison of choice Which is why I don't wanna be a devops. When we grow (if we grow) from the 3 people we are now, I totally will hire a devops whose job will be solely to deal with this poo poo. freeasinbeer posted:I’m spitballing ideas here, but maybe an image that pulls a docker container on startup from ECR and update the docker container. You’d probably also want to update the launch config with an explicit version. Now that you mention it, I just got an idea, please tell me if its really dumb: My application is one WAR file, one .SO (native library), and potentially 2 configuration files. The most often to update is the WAR file, less often the SO and the configuration files probably never. What if (is it even possible?) I put them in S3 when built and ready to go to production and I have the image pull them from S3 when starting up? Then I only have one AMI to use (that I prepare beforehand) and which will always launch the latest version of the app? And when I do the update I only have to tell the autoscaling group to launch a new instance and destroy the last one? Is that dumb? Can it work? freeasinbeer posted:I mean this is what heroku is for. But I don’t know a lot about the specifics here True, but we are in aws now (have domains and DNS there and load balancing) and heroku looked a bit too opaque (we also have native component to our app). I'm not maried with AWS, but changing ... is hard.
|
|
#
?
Apr 21, 2018 21:27
|
|
|
Volguus posted:Which is why I don't wanna be a devops. When we grow (if we grow) from the 3 people we are now, I totally will hire a devops whose job will be solely to deal with this poo poo. That s3 idea will work. Now someone should jump in and say it’s a bit risky in theory and that most huge places would avoid not explicitly setting the download artifact because you don’t know exactly what version is running, but in reality that should be fine for what you want.
|
|
#
?
Apr 21, 2018 22:23
|
|
|
Bootstrapping deployments from S3 on launch and just terminating your old instance every new version is perfectly fine. You still have to deal with OS patching because your AMI will be trapped in a past era. Creating a process for copying your launch configuration with the latest patch level base AMI and replacing that in your ASG would be good to do regularly so you don’t spend an hour patching your OS when that AMI gets very old is recommended. Certainly easier than having some AMI baking process for every release you do.
|
|
#
?
Apr 21, 2018 22:47
|
|
|
Volguus posted:Why ECS doesn't work? Because, as I said before, it takes 5 minutes for the thing to start up and launch the application that is in the container. I manually executed the lambda that started the task. So I believe you said you were using fargate for launching your task? i think you're hitting some issues around network interface connectivity when using public ips. if you set up a nat gateway in your vpc and don't enable public ip support for your fargate task, it should start up much faster and much more consistently (dependent on image size and application warmup).
|
|
#
?
Apr 21, 2018 23:08
|
|
|
FamDav posted:So I believe you said you were using fargate for launching your task? i think you're hitting some issues around network interface connectivity when using public ips. if you set up a nat gateway in your vpc and don't enable public ip support for your fargate task, it should start up much faster and much more consistently (dependent on image size and application warmup). Oh, that's interesting to hear. My application was not a service that would listen for connections on a port, I have absolutely no need to have a public IP, but without one it woudln't work (due to some other posts on the internet that I found, due to probably some other issues). Autoscaling though does work and I'm happy with it for now, except the hassle of updating the image. Arzakon posted:Bootstrapping deployments from S3 on launch and just terminating your old instance every new version is perfectly fine. You still have to deal with OS patching because your AMI will be trapped in a past era. Creating a process for copying your launch configuration with the latest patch level base AMI and replacing that in your ASG would be good to do regularly so you don’t spend an hour patching your OS when that AMI gets very old is recommended. Certainly easier than having some AMI baking process for every release you do. freeasinbeer posted:That s3 idea will work. Now someone should jump in and say it’s a bit risky in theory and that most huge places would avoid not explicitly setting the download artifact because you don’t know exactly what version is running, but in reality that should be fine for what you want. Cool, thanks for the confirmation. All I need is to hold me until we can get a real devops guy on board. My only other worry about S3 is if I can make it private (that is, only me from my own AWS network to access it).
|
|
#
?
Apr 22, 2018 00:19
|
|
|
Volguus posted:Cool, thanks for the confirmation. All I need is to hold me until we can get a real devops guy on board. My only other worry about S3 is if I can make it private (that is, only me from my own AWS network to access it). Yes, make sure the bucket/object policies only allow users from your account to access the bucket/objects, add an instance profile that can make S3 API calls to your instances, and use the aws cli in your userdata script to fetch the objects which will utilize the instance profile (don’t curl/wget). Since you mentioned network based restrictions you can do that in a bucket policy to where the bucket can only be accessed by IPs you designate or only from inside your VPC using the VPC Endpoint feature. That protects you in the case of someone releasing an access key into the wild on accident but only if that lost access key doesn’t have the rights to change the bucket policy. TLDR only use temporary access keys and make sure your poo poo is properly least privilege restricted.
|
|
#
?
Apr 22, 2018 00:37
|
|
|
Volguus posted:I totally will hire a devops whose job will be solely to deal with this poo poo.
|
|
#
?
Apr 22, 2018 15:10
|
|
|
StabbinHobo posted:god dammit 
|
|
#
?
Apr 22, 2018 15:16
|
|
|
Yeah I didn't want to call that out but 
|
|
#
?
Apr 22, 2018 15:58
|
|
|
Blinkz0rz posted:Yeah I didn't want to call that out but What did I do? Or say? Or ... how wrong was that sentence?
|
|
#
?
Apr 22, 2018 18:11
|
|
|
devops isn't a person, it's a methodology you wouldn't hire an agile person to agile up your software. altho i guess tons of places do this too
|
|
#
?
Apr 22, 2018 18:30
|
|
|
the talent deficit posted:devops isn't a person, it's a methodology Wait, devops is not a job title, a particular job description? "Devops" guys is totally I thing that I heard. And people do hire agile consultants, although I don't think many know what to actually expect of them. To me (and probably I'm wrong) devops is the team/guy who manages the build infrastructure, artifacts and deployment. Am I wrong to believe that?
|
|
#
?
Apr 22, 2018 18:33
|
|
|
|
| # ? Apr 25, 2024 18:21 |
|
|
Volguus posted:What did I do? Or say? Or ... how wrong was that sentence? saying "i'll just dump this mess on a devops janitor" is the perfectly distilled essence of not getting it and using a buzzphrase exactly 180 degrees backwards
|
|
#
?
Apr 22, 2018 18:38
|
|