Cloud Computing: Mostly fog machines in other people's datacenters

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Cloud Computing: Mostly fog machines in other people's datacenters

«‹›10 »

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Route53 alias record TTLs are set to the TTL of the target.

# ? Dec 23, 2015 14:35

Adbot: ADBOT LOVES YOU

# ? Apr 25, 2024 02:40

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

Tab8715 posted:

Extra-circular question but when it comes to massive web-based SaaS Applications like Facebook, Salesforce, Apple iCloud what are they using for their Directory Service?

This is specific for Salesforce, but everything I know about Salesforce's primary platform after talking to various employees over the years is that they actually don't even have that many servers (< 1500) and that they don't even use virtualization, so I'd make a guess that software-wise they're still probably fine with LDAP. I know they aren't using CA SiteMinder (the only other alternative to Active Directory worth a drat in any fashion that gives consideration to dinosaur enterprise needs and enterprise feature check boxes). And to add to the above, Salesforce's primary tech stack was based around hiring Oracle engineers (and Benioff is ex-Oracle himself after all) so they went with Oracle because it made the most sense for them. Salesforce's scaling problems are almost nothing like the problems that face most b2c platforms and will have a lot more concerns about regulatory considerations for their customers that make dealing with oftentimes tech-clueless regulators easiest with just pulling crap into a traditional enterprise RDBMS like Oracle and just treat it like a typical ETL platform with data warehousing that's the bread and butter of Oracle's business.

# ? Dec 26, 2015 06:12

MagnumOpus: Dec 7, 2006

Currently sitting here with my thumb up my butt, waiting for a call back from my internal Openstack provider. I needed some more resources so I went into their customized Horizon portal and configured a quota increase request. Clicked Submit, got an error page, and then 30 seconds later an email letting me know my request to deactivate all of our configured user accounts, including the ones for automation, had completed successfully.

:bang:

# ? Jan 7, 2016 17:42

in a well actually: Jan 26, 2011; dude, you gotta end it on the rhyme

MagnumOpus posted:

Currently sitting here with my thumb up my butt, waiting for a call back from my internal Openstack provider. I needed some more resources so I went into their customized Horizon portal and configured a quota increase request. Clicked Submit, got an error page, and then 30 seconds later an email letting me know my request to deactivate all of our configured user accounts, including the ones for automation, had completed successfully.

@openstack_txt

# ? Jan 7, 2016 20:17

Wicaeed: Feb 8, 2005

jesus

# ? Jan 9, 2016 10:26

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

Sounds like someone customized that Horizon portal..... poorly.

# ? Jan 12, 2016 19:39

Ixian: Oct 9, 2001; Many machines on Ix....new machines; Pillbug

Sums up Openstack for me (it does everything you need! Everything! Just...badly. A script will fix it, most of the time!)

Find a good vodka you like and hunker down, it's gonna be a long night.

# ? Jan 13, 2016 05:03

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

If my OpenStack servers weren't blades I would literally pull them out from the rack on their rails and poo poo inside them

# ? Jan 13, 2016 06:33

Bhodi: Dec 9, 2007; Oh, it's just a cat.; Pillbug

I am tired of the cloud today.

We just fired another RH consultant for underperforming, again. One of our devs goes in to try and untangle the mess he left behind and the code is a horror show of cut-pasted together crap with other people's names all over it, big surprise

At least i'm not in our sister group, the openstack team lost a drive over the weekend and an important project's VMs were lost and so their director got pulled in and reamed out hard on Monday by senior leadership for the fuckup (if not directly by misconfiguration / poor architecture on his group's end, than failure to communicate the importance of availability zones and risk of provisioning to users)

:yayclod:

Bhodi fucked around with this message at 18:06 on Jan 13, 2016

# ? Jan 13, 2016 18:02

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

Wait... no backups?

Although ironically, my group trying to setup backups for our Chef installation (less than 10k nodes across thousands of users and people think we're getting overloaded first thing, lol) has caused more serious outages and failures than if we had never bothered with back-ups. Hell, we had Netbackup running (which was causing some of the outages, ironically too) that could have restored our systems, so we had double backups going for a while. The same thing goes for our Chef HA setup (it's caused more availability loss than anything due to split brain problems happening with a really, really flaky ESX cluster running way, way overprovisioned on everything when we randomly drop ICMP packets for a few times in a row even though no vMotions are happening).

# ? Jan 13, 2016 19:44

Aunt Beth: Feb 24, 2006; Baby, you're ready!; Grimey Drawer

Vulture Culture posted:

If my OpenStack servers weren't blades I would literally pull them out from the rack on their rails and poo poo inside them

Is the point that you'd poo poo in them while they're still running? Because I mean, dropping small ones into blades would be pretty great. Chassis' air movers are mighty powerful. Could be fun.

# ? Jan 13, 2016 20:07

bitprophet: Jul 22, 2004; Taco Defender

How does a single drive failure cause any data loss in TYOOL 2016? :psyduck:

# ? Jan 13, 2016 20:31

Ixian: Oct 9, 2001; Many machines on Ix....new machines; Pillbug

bitprophet posted:

How does a single drive failure cause any data loss in TYOOL 2016?

Because open stack is the Perl of cloud computing

# ? Jan 13, 2016 22:30

jre: Sep 2, 2011; To the cloud ?

bitprophet posted:

How does a single drive failure cause any data loss in TYOOL 2016?

OpenStack is an amazing machine made of super complex glass parts spinning at 100,000 rpm. Don't Sneeze

Oh gently caress you looked at it wrong and theres glass everywhere

# ? Jan 14, 2016 01:14

evol262: Nov 30, 2010; #!/usr/bin/perl

Ixian posted:

Because open stack is the Perl of cloud computing

It's because their admins didn't bother configuring a resilient backend for glance, or their storage policies weren't set up (or were misconfigured) for swift.

Openstack isn't even remotely perfect, and it's confusing for admins, but it probably doesn't have the blame for this.

As an aside, Perl is great. Call Openstack the PHP or Node.js if you want, but Perl built the world.

# ? Jan 14, 2016 01:58

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

evol262 posted:

It's because their admins didn't bother configuring a resilient backend for glance, or their storage policies weren't set up (or were misconfigured) for swift.

Or the cluster is set up with instance storage by default, which is sensible for many environments. AWS does the same thing if you launch an instance store AMI.

# ? Jan 14, 2016 02:24

evol262: Nov 30, 2010; #!/usr/bin/perl

Vulture Culture posted:

Or the cluster is set up with instance storage by default, which is sensible for many environments. AWS does the same thing if you launch an instance store AMI.

But then we have to assume that all the VMs were running on a single compute node, and that a single disk failure killed that node (or at least /var), which is even dumber.

# ? Jan 14, 2016 02:37

Ixian: Oct 9, 2001; Many machines on Ix....new machines; Pillbug

evol262 posted:

It's because their admins didn't bother configuring a resilient backend for glance, or their storage policies weren't set up (or were misconfigured) for swift.

Openstack isn't even remotely perfect, and it's confusing for admins, but it probably doesn't have the blame for this.

As an aside, Perl is great. Call Openstack the PHP or Node.js if you want, but Perl built the world.

Perl IS great. The glue that still holds a lot of the net together, no doubt.

In the right hands it is incredibly powerful and works wonders. However you really have to know what the gently caress you are doing and/or not push it too far out of bounds. Perl is great but for every Perl master there's a dozen neophytes out of their league setting the stage for long term problems because at first glance what they did "seemed to work fine" or they are just running something someone else did with no real idea how it works or desire to know why.

.....just like Openstack.

# ? Jan 14, 2016 02:37

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

evol262 posted:

But then we have to assume that all the VMs were running on a single compute node, and that a single disk failure killed that node (or at least /var), which is even dumber.

Dumber in the sense that people are obviously doing the wrong thing with it (either expectations haven't been communicated, or people aren't listening regardless), but there's nothing at all wrong with this approach for cattle when you're running OpenStack as designed. The instances blow up, and Heat spins new ones. Why double your disk costs for no reason? If you need individual instances to be resilient against disk failures, you have Cinder.

# ? Jan 14, 2016 03:24

evol262: Nov 30, 2010; #!/usr/bin/perl

Ixian posted:

Perl IS great. The glue that still holds a lot of the net together, no doubt.

In the right hands it is incredibly powerful and works wonders. However you really have to know what the gently caress you are doing and/or not push it too far out of bounds. Perl is great but for every Perl master there's a dozen neophytes out of their league setting the stage for long term problems because at first glance what they did "seemed to work fine" or they are just running something someone else did with no real idea how it works or desire to know why.

.....just like Openstack.

I'd agree and disagree. I agree in a general sense that the basic tenets of cloud computing (how to structure your applications, how to handle resiliency, etc) are a quagmire for people who just want to get something up and running, no matter whether it's GCE, AWS, Openstack, or another provider. Openstack is harder because you actually need to configure Openstack itself (correctly), which has a whole bunch of problems because the architecture (in particular, the overly chatty applications hammering the database instead of using a message queue) falls over in large environments unless you have subject matter experts.

But "I lost a disk, and a server crashed, with all my stuff on it!" (or "I lost a disk, and my entire VMware datastore was on it!" [Glance], or "I lost a disk, and my RAID failed!" [bad swift resiliency]) isn't Openstack's fault.

I'm not really trying to defend Openstack here, and I know as well as anybody that it's just not appropriate for 95% of the people who want to use it, and 50% of client who it's appropriate for don't have the on-site expertise to keep it ticking, but losing a disk shouldn't cause significant damage to an Openstack environment with an admin who can rub two brain cells together. It's :yaycloud:

, but it's not magic. The cloud is just someone else's computer (yours, in the case of openstack), and all the hard-learned lessons about how to make sure your environment doesn't take a dive when you lose a disk or a network link or whatever still apply.

Vulture Culture posted:

Dumber in the sense that people are obviously doing the wrong thing with it (either expectations haven't been communicated, or people aren't listening regardless), but there's nothing at all wrong with this approach for cattle when you're running OpenStack as designed. The instances blow up, and Heat spins new ones. Why double your disk costs for no reason? If you need individual instances to be resilient against disk failures, you have Cinder.

I try not to have customer involvement, but almost everyone sosreports from I've seen is still running on PowerEdge/Proliant/whatever with at least two disks in fault-tolerant RAID.

Sure, it doubles your disk costs, but most of our institutional customers seem to have accepted that as a cost of business for years, and haven't changed their purchasing strategy just for compute nodes. They'd probably be better off shoving as much memory and as little disk as possible into the chassis, but never my call.

I'm also guessing from the total loss of the VMs (and that they cared) that these were probably pets.

evol262 fucked around with this message at 03:34 on Jan 14, 2016

# ? Jan 14, 2016 03:30

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

Half the places I've seen using OpenStack come from a VMware-ish or traditional infrastructure history where you're used to treating every host as a pet, so everyone should be running high availability storage and network to some degree as a habit. If people are starting to deploy OpenStack thinking it magically fixes all these things for you, I dunno if you just became brain damaged or what.

# ? Jan 14, 2016 05:34

MagnumOpus: Dec 7, 2006

Wait wait wait are you telling me that I can't just shove a 3-node cassandra cluster into my environment, try to hack it into performing like an immediately-consistent database, and expect the cloud to magically recover everything when I lose one of those nodes.

# ? Jan 14, 2016 15:13

Bhodi: Dec 9, 2007; Oh, it's just a cat.; Pillbug

OSes are magic, virtualization is magic, the cloud is magic, everything is magic

Sadly, magic doesn't mean operable, or good

# ? Jan 14, 2016 17:04

MagnumOpus: Dec 7, 2006

Bhodi posted:

OSes are magic, virtualization is magic, the cloud is magic, everything is magic

Sadly, magic doesn't mean operable, or good

I feel like this is the biggest change that occurred in the web ops world in terms of its effect on my day-to-day. It used to be that no one else in the org cared about what we were doing on an infrastructure level because they knew it was a nightmare realm they dared not enter. Since the rise of everything "cloud" now we've got armies of web devs who know just enough to be dangerous. My job is now running around trying to keep assholes from loving up long-term plans by applying half-understood cloudisms to their designs. Just about every day I find a new thing that makes me go cross-eyed with rage, and on the days I don't it's because instead I spent 3 hours arguing a web dev down from his 20% understanding of platform and infrastructure concepts.

# ? Jan 14, 2016 17:54

Pile Of Garbage: May 28, 2007

Azure question: is it possible to override system routes on the platform? For example, we've deployed a Check Point appliance which we want to act as the default gateway for all VMs so that it can enforce policy and inspect traffic. However we've only managed to get it to work for VM traffic going between subnets. Traffic going from a VM to a VPN gateway or the internet just magically bypasses the Check Point appliance (Which I assume is due to SDN voodoo).

Is this at all possible?

# ? Jan 14, 2016 18:49

Gucci Loafers: May 20, 2006; Can't post for 5 hours!

cheese-cube posted:

Azure question: is it possible to override system routes on the platform? For example, we've deployed a Check Point appliance which we want to act as the default gateway for all VMs so that it can enforce policy and inspect traffic. However we've only managed to get it to work for VM traffic going between subnets. Traffic going from a VM to a VPN gateway or the internet just magically bypasses the Check Point appliance (Which I assume is due to SDN voodoo).

Is this at all possible?

Does this help?

Azure User-Defined Routes

# ? Jan 14, 2016 20:21

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

MagnumOpus posted:

I feel like this is the biggest change that occurred in the web ops world in terms of its effect on my day-to-day. It used to be that no one else in the org cared about what we were doing on an infrastructure level because they knew it was a nightmare realm they dared not enter. Since the rise of everything "cloud" now we've got armies of web devs who know just enough to be dangerous. My job is now running around trying to keep assholes from loving up long-term plans by applying half-understood cloudisms to their designs. Just about every day I find a new thing that makes me go cross-eyed with rage, and on the days I don't it's because instead I spent 3 hours arguing a web dev down from his 20% understanding of platform and infrastructure concepts.

This is a great reason to have devs responsible for operating their own software

# ? Jan 14, 2016 20:38

MagnumOpus: Dec 7, 2006

Vulture Culture posted:

This is a great reason to have devs responsible for operating their own software

O-ho-ho but you see "we are all just engineers in this group" because we're "avoiding silo-ization". We're "flat" and a "unified development team" "leveraging devops paradigms" and responsible for "self-organizing".

EDIT: All of these are real things that can be accomplished. I don't work for a ~~company~~ group that understands how to do that, so instead we just say pay lip service to all this except for using it to chastise people who point out problems.

MagnumOpus fucked around with this message at 21:00 on Jan 14, 2016

# ? Jan 14, 2016 20:47

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

I've learned to just say "Yes, we can do it, but... here's what you're trading off" and documenting it carefully while I silently work on counter-measures / contingencies for the inevitable failure of the bad design. If a place can't get a lot of system design right, there's a fair chance they're not going to get even basic stuff like HA network and storage right and you're just on your own really. Replication factor of 1? Absolutely fine for your goal of cost savings, Mr. SVP! I'll check with the backup guys that we've put all these into backup inclusions in the meantime if that's alright with you?

Vulture Culture posted:

This is a great reason to have devs responsible for operating their own software

Yeah, and this is why I wound up doing ops in the end for a software team because I had to make up for all the other software engineers that couldn't wrap their heads (and more importantly - didn't want to care) around why you shouldn't just run JBODs, unmask all LUNs on the SANs, stop IPtables, disable SELinux and AppArmor on a public-facing web server, or have flat 10.0.0.0/8 subnets everywhere. Going fast by removing safety measures seems to be fashionable among developers that don't care because "I made it compile and pass my meaningless unit tests with like 20% coverage" is the modus operandi of developer-centric start-ups and because they're paid stupidly high wages they feel they're an authority on anything besides just plain code. Nowadays, I'm now banging my head against ops engineers that don't understand software from this century and how to plan infrastructure for "all these newfangled clouds."

Also, gently caress the developers that think reading highscalability.com posts means they're a goddamn CCIE and VCDX architect.

# ? Jan 14, 2016 23:02

MagnumOpus: Dec 7, 2006

necrobobsledder posted:

developer-centric start-ups and because they're paid stupidly high wages they feel they're an authority on anything besides just plain code

Also, gently caress the developers that think reading highscalability.com posts means they're a goddamn CCIE and VCDX architect.

These are unfortunately the roots of my problems. My company is huge and developer-centric, and the business unit I'm in has only ever built software that is run on-premise by their customers so they have absolute zero experience in running a webapp enterprise. This fact has not stopped their "architects" from making designs that commit us to poorly-researched solutions before the ops team has any opportunity to intervene. For examples they were recently stumped by the eventually-consistent nature of Cassandra, forcing a massive refactor of their app, and don't even get the PM started on how badly they missed the mark on estimating SSD storage costs.

# ? Jan 14, 2016 23:14

Pile Of Garbage: May 28, 2007