Cloud Computing: Mostly fog machines in other people's datacenters

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Cloud Computing: Mostly fog machines in other people's datacenters

«‹›10 »

Virigoth: Apr 28, 2009; Corona rules everything around me
C.R.E.A.M. get the virus
In the ICU y'all......

Has anyone here used Spinnaker and has thoughts on it? Our system is a Jenkins master right now that supports 3 lower environments for dev, QA, and our staging environment and then 4 production regions in AWS. We managed around 2,000 servers for micro-services but are growing daily. Our Jenkins master does about 16,000 jobs a day and uses a in-house jslave script to deploy the builds. With the permissions needed to deploy this is a scary scenario. What I'm considering doing a proof of concept on is hiding Jenkins behind Spinnaker and using it just like the demo video sets up with Spinnaker being our front end and a lot of user access removed.

# ? Mar 16, 2016 02:29

Adbot: ADBOT LOVES YOU

# ? Apr 23, 2024 10:37

fluppet: Feb 10, 2009

I'd love to give spinnaker a shot but I'm currently at a place that's stuck with teamcity and haven't been able to get the pair to play nice

If you do get it working can we get a trip report?

# ? Mar 20, 2016 14:41

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Can I create a big pile of stopped instances on EC2 without powering them on (and incurring the hourly charge) first?

# ? Apr 15, 2016 18:42

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

I don't think that's possible. I know I can do it with the API in VMware's stuff, but in EC2 you have to launch an instance to create or clone one, and launching means it gets put into pending and then running state. The official lifecycle document from AWS pretty much means that you don't get any state of an instance before the Pending state. http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-lifecycle.html Only thing I could think of as a sneaky way possibly is to create an AMI that will immediately shut down the machine before the bootloader kicks in the first time you launch it, but I suspect that won't help because it might have to be put into the Running state first for that to kick in.

If you need to queue up a bunch of instances to be able to handle something like a spike load it's easy to forget that you have to let AWS know so that your ELBs don't get run over by a freight train. I think that applies for even internal ELBs.

# ? Apr 15, 2016 23:05

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

necrobobsledder posted:

I don't think that's possible. I know I can do it with the API in VMware's stuff, but in EC2 you have to launch an instance to create or clone one, and launching means it gets put into pending and then running state. The official lifecycle document from AWS pretty much means that you don't get any state of an instance before the Pending state. http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-lifecycle.html Only thing I could think of as a sneaky way possibly is to create an AMI that will immediately shut down the machine before the bootloader kicks in the first time you launch it, but I suspect that won't help because it might have to be put into the Running state first for that to kick in.

If you need to queue up a bunch of instances to be able to handle something like a spike load it's easy to forget that you have to let AWS know so that your ELBs don't get run over by a freight train. I think that applies for even internal ELBs.

That's not a concern, there's no load balancers involved on these. I just want some people here to be able to power servers on to add capacity without doing anything funny (auto-scaling groups are a really poor fit for stateful services).

# ? Apr 16, 2016 00:11

Agrikk: Oct 17, 2003; Take care with that! We have not fully ascertained its function, and the ticking is accelerating.

Vulture Culture posted:

Can I create a big pile of stopped instances on EC2 without powering them on (and incurring the hourly charge) first?

It is not possible.

The best you can do is fire it up, let it bake and then shut it down. You will incur an hour charge for each one, because AWS charges by the hour.

If you have the time, try bidding on the spot market for cheap prices. Maybe that will help keep the cost down somewhat.

# ? Apr 18, 2016 04:38

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

I'm firing these up in bulk through Terraform, so there's no big deal if I have to actually start them in response to demand. It would have been nice to be able to hand our CTO or whoever instructions to just power a bunch of stuff on, though.

(ASGs don't work for our use case for a litany of reasons I'm not going to get into.)

# ? Apr 18, 2016 06:07

Internet Explorer: Jun 1, 2005

Hopefully quick HIPAA / Cloud question. I don't deal with HIPAA but something I heard does not pass the smell test.

If a cloud provider has admin access to a Windows VM on their infrastructure, is it possible for them to be HIPAA complaint? I find a hard time believing that they would be without going through the same paperwork required to share HIPAA data from the owners of that data.

# ? May 18, 2016 19:30

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Internet Explorer posted:

Hopefully quick HIPAA / Cloud question. I don't deal with HIPAA but something I heard does not pass the smell test.

If a cloud provider has admin access to a Windows VM on their infrastructure, is it possible for them to be HIPAA complaint? I find a hard time believing that they would be without going through the same paperwork required to share HIPAA data from the owners of that data.

There's a standard for transfer of liability which is covered under the Business Associate Contracts section, so it's very possible to be HIPAA compliant if the contract language meets the standards under �164.314(a)(1). There may be specific implementation details which don't pass the smell test in other ways. For example, "a cloud provider has admin access" sounds to me like they have a single default admin account which would not fulfill the HIPAA data access auditing requirements. If a business is aware of ways the Business Associate/Covered Entity is not holding up their end of the data privacy standards, there is specific language in �164.314(a)(1)(ii) determining how an organization should address that to avoid their own complicity/liability.

Vulture Culture fucked around with this message at 19:54 on May 18, 2016

# ? May 18, 2016 19:51

Internet Explorer: Jun 1, 2005

Thank you for the very detailed post. I am not involved on the technical side, so I only have limited information, but I believe it is a generic admin account which I know is a no-no. I know that there is 2FA that can tie a 2FA key to a specific user when using a generic account, but that seems somewhat sketchy as well if sensitive information is involved.

This same provider is doing a lot of other dumb things, so I wouldn't be surprised if they are not following guidelines. poo poo like giving 16 vCPU to VMs by default, not applying Windows updates (and when they do, skipping Service Packs), etc. Thanks again!

# ? May 18, 2016 20:06

incoherent: Apr 24, 2004; 01010100011010000111001
00110100101101100011011
000110010101110010

You need to enable fine grain billing if you're having a provider handle your cloud migration or prem2cloud expansion (if thats what you're doing). If you're not careful you can blow alotta cash easily.

# ? May 18, 2016 21:36

Internet Explorer: Jun 1, 2005

incoherent posted:

You need to enable fine grain billing if you're having a provider handle your cloud migration or prem2cloud expansion (if thats what you're doing). If you're not careful you can blow alotta cash easily.

Thankfully, this is not for me and I do not handle HIPAA-covered data. Was just asking for a friend. They are small non-profit involved in healthcare and don't really have someone with IT chops. A great combo!

# ? May 18, 2016 22:34

good jovi: Dec 11, 2000; 'm pro-dickgirl, and I VOTE!

Does anyone here have any experience setting up a VPN endpoint in an AWS VPC? Everything I've been able to find seems to be aimed at site-to-site connections, rather than just something for developers to connect to. It looks like this involves running some 3rd party software appliance, rather than being built in to AWS itself. Any recommendations there?

# ? May 20, 2016 19:15

Bhodi: Dec 9, 2007; Oh, it's just a cat.; Pillbug

VPC *IS* for site to site connections. It's when you need to have IP namespace within your organization. It's not designed to be used like you're trying to use it, which is why you're looking at cobbled together solutions. You're using the wrong tool for your particular job.

# ? May 20, 2016 22:09

Thanks Ants: May 21, 2004; #essereFerrari

Either run the VPN back to your office and terminate client VPN there, or deploy one of the SSL VPN virtual appliances into your VPC.

# ? May 20, 2016 22:15

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Bhodi posted:

VPC *IS* for site to site connections.

No it isn't. Are you confusing the VPC with the Virtual Private Gateway?

good jovi posted:

Does anyone here have any experience setting up a VPN endpoint in an AWS VPC? Everything I've been able to find seems to be aimed at site-to-site connections, rather than just something for developers to connect to. It looks like this involves running some 3rd party software appliance, rather than being built in to AWS itself. Any recommendations there?

Correct, the VPC itself does not supply a VPN gateway, and the Virtual Private Gateway product is oriented towards IPsec-only site-to-site connections. If you want to connect up random endpoints, you'll need something that supports L2TP+IPsec or PPTP. I've used Openswan in the past with some fiddling, but nowadays I'd probably use a VyOS (Vyatta fork) appliance or something else that handles the Openswan/Strongswan+xl2tpd+pppd stuff a little more transparently. If you control the endpoints, you might want to just use something easy like OpenVPN. And you'll want to turn off Source/Destination Check on your instance, just like with a NAT gateway on your VPC, or you won't be able to push traffic back out through it to your clients.

Vulture Culture fucked around with this message at 01:24 on May 21, 2016

# ? May 21, 2016 01:19

MrMoo: Sep 14, 2000

LT2P is no longer needed, just use StrongSwan and IKEv2.

# ? May 21, 2016 17:09

MF_James: May 8, 2008; I CANNOT HANDLE BEING CALLED OUT ON MY DUMBASS OPINIONS ABOUT ANTI-VIRUS AND SECURITY. I REALLY LIKE TO THINK THAT I KNOW THINGS HERE

INSTEAD I AM GOING TO WHINE ABOUT IT IN OTHER THREADS SO MY OPINION CAN FEEL VALIDATED IN AN ECHO CHAMBER I LIKE

Yeah you can be compliant in HIPAA, JSOX and PCI in cloud datacenters, just bring that up during contract negotiations and have specific words written into the contract, aka talk to legal and have them figure it out.

# ? May 21, 2016 17:17

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

MrMoo posted:

LT2P is no longer needed, just use StrongSwan and IKEv2.

True, IKEv2 is a much simpler option nowadays and provides a much better experience to people hopping around on wifi/LTE/etc.

# ? May 21, 2016 18:25

Virigoth: Apr 28, 2009; Corona rules everything around me
C.R.E.A.M. get the virus
In the ICU y'all......

OpenVPN is hard to manage and has awful support. Rebooting can cause licenses to become detached and the turn around time is about 3 days. We just switched to softether + onelogin with an ansiblized deployment package and it works great for us so far.

I'm on phone but can go into openvpn woes for 400+ users(100 concurrent or so)in AWS later if you want more details.

# ? May 21, 2016 19:17

KillHour: Oct 28, 2007

Is there a way to easily snapshot machines in Azure? My company decided I needed to start using :yayclod:

for training because it's the future, so I'm sitting here staring at an Azure dashboard like "WTF do I do now?"

My use case is this:

I need each person in a training (typically 4-12 people) to have their own environment with (at a minimum) 3 VMs. These VMs should start in a mostly empty state with some minor preconfiguration done. I then walk them through installing each of the components on each of their VMs, which will be communicating with one another on a private virtual network. After they have done all the configuration on all the machines, I need to be able to revert all of their VMs to that default preconfigured state, just the way I had it for the next class. I also need to create snapshots along the way so if a student messes up somehow and breaks everything, I can pull the cooking show "cut ahead to the next step" for them. I need to be able to do all this seamlessly and with no more than about 5 minutes of downtime because class is pretty tight as it is, so I imagine I need to write a powershell script to handle this for me. I just have no idea where to start since I used to do classes on a VMWare server I would bring with me and that's all I really know how to use.

# ? Aug 11, 2016 06:15

Gucci Loafers: May 20, 2006; Ask yourself, do you really want to talk to pair of really nice gaudy shoes?

KillHour posted:

Is there a way to easily snapshot machines in Azure? My company decided I needed to start using for training because it's the future, so I'm sitting here staring at an Azure dashboard like "WTF do I do now?"

My use case is this:

I need each person in a training (typically 4-12 people) to have their own environment with (at a minimum) 3 VMs. These VMs should start in a mostly empty state with some minor preconfiguration done. I then walk them through installing each of the components on each of their VMs, which will be communicating with one another on a private virtual network. After they have done all the configuration on all the machines, I need to be able to revert all of their VMs to that default preconfigured state, just the way I had it for the next class. I also need to create snapshots along the way so if a student messes up somehow and breaks everything, I can pull the cooking show "cut ahead to the next step" for them. I need to be able to do all this seamlessly and with no more than about 5 minutes of downtime because class is pretty tight as it is, so I imagine I need to write a powershell script to handle this for me. I just have no idea where to start since I used to do classes on a VMWare server I would bring with me and that's all I really know how to use.

Use Azure Backup in ARM. Essentially, you'll create a backup of each Virtual Machine then if something messes up or you need to go back to the default configuration delete the original VM then re-deploy from the backup.

Granted, this won't be a quick process and you could write a very dense Powershell Script to take snapshots of the page blob but that's going to very complex. Additionally, the backups aren't quick either and you might want to use SSDs but again that costs money.

# ? Aug 11, 2016 06:33

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Speaking of Azure, I have a Packer image of Ubuntu 16.04 that doesn't clone correctly. I get:

code:

Provisioning failed. OS Provisioning for VM 'go-gently caress-yourself-vulture-culture' did not finish in the
allotted time. However, the VM guest agent was detected running. This suggests the guest
OS has not been properly prepared to be used as a VM image (with CreateOption=FromImage).
To resolve this issue, either use the VHD as is with CreateOption=Attach or prepare it properly
for use as an image:
 * Instructions for Windows: [url]https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-upload-image/[/url] 
 * Instructions for Linux: [url]https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-capture-image/.[/url] OSProvisioningTimedOut

Is there any way to see what's actually failing validation, or am I just stuck with a typical Microsoft "try every one of these hundred things and maybe one of them will fix this error message" troubleshooting workflow?

# ? Aug 11, 2016 06:38

Gucci Loafers: May 20, 2006; Ask yourself, do you really want to talk to pair of really nice gaudy shoes?

Vulture Culture posted:

Speaking of Azure, I have a Packer image of Ubuntu 16.04 that doesn't clone correctly. I get:
code:
Provisioning failed. OS Provisioning for VM 'go-gently caress-yourself-vulture-culture' did not finish in the
allotted time. However, the VM guest agent was detected running. This suggests the guest
OS has not been properly prepared to be used as a VM image (with CreateOption=FromImage).
To resolve this issue, either use the VHD as is with CreateOption=Attach or prepare it properly
for use as an image:
 * Instructions for Windows: [url]https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-upload-image/[/url] 
 * Instructions for Linux: [url]https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-capture-image/.[/url] OSProvisioningTimedOut
Is there any way to see what's actually failing validation, or am I just stuck with a typical Microsoft "try every one of these hundred things and maybe one of them will fix this error message" troubleshooting workflow?

Where are you seeing the error? Are deploying with Azure Powershell?

# ? Aug 11, 2016 06:44

KillHour: Oct 28, 2007

Tab8715 posted:

Use Azure Backup in ARM. Essentially, you'll create a backup of each Virtual Machine then if something messes up or you need to go back to the default configuration delete the original VM then re-deploy from the backup.

Granted, this won't be a quick process and you could write a very dense Powershell Script to take snapshots of the page blob but that's going to very complex. Additionally, the backups aren't quick either and you might want to use SSDs but again that costs money.

So that's a problem because I don't need just a default configuration. I need 4-5 different "states" depending on where we are in class (e.g.: SQL installed but database not yet created), and I need to be able to get that student there in a matter of minutes. In VMware, I would just right click their VM, go to snapshot manager, and pick the correct state. As soon as the student could remote back in, the machine was up and running. I need similar functionality here.

Also, after every class, I need the ability to reset a number of VMs (Typically somewhere between 9 and 39) to their default state and have them boot up and be ready for me. Again, that was trivial with a short VMware script.

# ? Aug 11, 2016 06:45

Gucci Loafers: May 20, 2006; Ask yourself, do you really want to talk to pair of really nice gaudy shoes?

Take a backup for each desired configuration.

# ? Aug 11, 2016 06:59

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Tab8715 posted:

Where are you seeing the error? Are deploying with Azure Powershell?

Terraform. The error is in the Azure portal.

# ? Aug 11, 2016 13:26

Ryaath: Apr 8, 2003

Anyone have any experience deploying and managing an open source Cloud Foundry? Any pitfalls I should avoid? Ours is up and running on Openstack with minimal effort. We even built a concourse thingy for CI/CD so updates seem to be non events... I'm impressed so far with zero apps/users...

# ? Aug 12, 2016 01:00

Gucci Loafers: May 20, 2006; Ask yourself, do you really want to talk to pair of really nice gaudy shoes?

Vulture Culture posted:

Terraform. The error is in the Azure portal.

Their program is calling the incorrect APIs - something's up.

# ? Aug 12, 2016 01:09

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Tab8715 posted:

Their program is calling the incorrect APIs - something's up.

The APIs are fine. I'm certain something is up with the image, which works in a very non-standard way (think CoreOS-style system running on a read-only overlay filesystem), but Azure makes it really hard for me to figure that out without giving me so much as waagent logs or anything.

# ? Aug 12, 2016 02:42

Gucci Loafers: May 20, 2006; Ask yourself, do you really want to talk to pair of really nice gaudy shoes?

Vulture Culture posted:

The APIs are fine. I'm certain something is up with the image, which works in a very non-standard way (think CoreOS-style system running on a read-only overlay filesystem), but Azure makes it really hard for me to figure that out without giving me so much as waagent logs or anything.

As far as I know - the Portal is claiming your trying to deploy this image in a way that isn't supported. Try hitting out to Azure Twitter and you should be able to open a free support case.

# ? Aug 12, 2016 21:24

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Tab8715 posted:

As far as I know - the Portal is claiming your trying to deploy this image in a way that isn't supported. Try hitting out to Azure Twitter and you should be able to open a free support case.

Support wasn't super-helpful on this, but the issue was that the Azure platform won't try to provision if you aren't forwarding the console to the serial port. This is documented precisely nowhere.

# ? Aug 22, 2016 21:57

incoherent: Apr 24, 2004; 01010100011010000111001
00110100101101100011011
000110010101110010

What? Cloud based serial port?

# ? Aug 22, 2016 22:20

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

incoherent posted:

What? Cloud based serial port?

Serial console is a very common way of getting Linux startup logs on basically any cloud platform (e.g. EC2, GCE), but I've never heard of a cloud system outright failing if you don't have it enabled. It must check for something in the logs before it starts to run the provisioner piece that talks to the Azure agent (a component which is, in and of itself, another huge WTF).

# ? Aug 22, 2016 23:59

Gucci Loafers: May 20, 2006; Ask yourself, do you really want to talk to pair of really nice gaudy shoes?

Vulture Culture posted:

Serial console is a very common way of getting Linux startup logs on basically any cloud platform (e.g. EC2, GCE), but I've never heard of a cloud system outright failing if you don't have it enabled. It must check for something in the logs before it starts to run the provisioner piece that talks to the Azure agent (a component which is, in and of itself, another huge WTF).

I guess the documentation doesn't say it's required. On the other hand, everything you want to know about the WALA is here.

# ? Aug 23, 2016 05:40

Michaellaneous: Oct 30, 2013

Not sure if this questions belongs here, but it's OpenStack so?

What the gently caress is the functional difference of Object Storage and Block Storage in OS? What data is stored on each of the systems? Why does OS suggest having both at the same time?

# ? Aug 23, 2016 12:28

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Michaellaneous posted:

Not sure if this questions belongs here, but it's OpenStack so?

What the gently caress is the functional difference of Object Storage and Block Storage in OS? What data is stored on each of the systems? Why does OS suggest having both at the same time?

Oversimplifying a little bit:

Swift is analogous to Amazon S3. It's buckets of storage for random blobs of data that don't need random seeks within the file. Because it doesn't have any need for POSIX I/O semantics, you can trivially do things like shard a file between a number of servers and then replicate it several times for redundancy without your clients needing to understand anything more complex than HTTP. Cinder is analogous to Amazon EBS: it's for storing volumes. Cinder is backend-agnostic, so it can be backed by LVM, or a SAN, or Ceph, or whatever your little heart desires (as long as there's a backend driver for it).

# ? Aug 23, 2016 14:21

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

Even more roughly, block storage is meant to be used like it's attached to another, mutually exclusive device (usually single entity like a virtual machine), and object storage is generally expected for use from remote locations and should support access semantics appropriate for those use cases. At least this was sufficient enough for people that aren't familiar with AWS in my past.

# ? Aug 23, 2016 14:48

DevNull: Apr 4, 2007; And sometimes is seen a strange spot in the sky
A human being that was given to fly

Someone's script being able to just disable VMs doesn't seem like a good place to be. http://www.fredtrotter.com/2016/08/22/google-intrusion-detection-problems/

# ? Aug 23, 2016 15:42

Adbot: ADBOT LOVES YOU

# ? Apr 23, 2024 10:37

Potato Salad: Oct 23, 2014; nobody cares

What else are they gonna do?

# ? Aug 23, 2016 16:04

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Cloud Computing: Mostly fog machines in other people's datacenters

«‹›10 »