Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Bhodi
Dec 9, 2007

Oh, it's just a cat.
Pillbug

Negromancer posted:

The best thing to take away from this is gently caress iptables. As long as you have your security groups setup correctly, you really should not need to run a software firewall.


Yea, trying to run a DB on a ec2 instance is terrible. That is the reason they have RDS offering now, but that is still only oracle/mysql/MSSQL only. I think they are going to start offering postgres eventually, but god knows when. You could probably get away with running postgres on one of the super huge EBS optimized instances, but they cost so much its really not worth it.

As much as I love AWS, its not the solution for everything and you really need to architect everything around its limitations. Granted, I mostly love running everything at my current job out of AWS since it means no datacenter to deal with.

The future is now!

Adbot
ADBOT LOVES YOU

Bhodi
Dec 9, 2007

Oh, it's just a cat.
Pillbug
I had to explain to some super-aggressive DBAs in a meeting that while I supported their log-scrape witch hunt to discover why oracle poo poo the bed over the weekend, they simply could not forward me 5 pages of /var/log/messages starting 30 minutes before the incident, with dozens of lines they thought "were concerning" bolded and underlined.

Lines that included our puppet runs updating various unrelated files on the system and a ntpd restart 25 minutes before the crash. They demanded I explain line by line what each item meant and also provide documentation on everything puppet does so they could examine it for "any possible conflicts".

I told them that puppet runs enterprise wide and has been running for over 2 years without issue. I then, sadly, made a cardinal sin: I told them that oracle was simply another application, it was not special, and that while it may have more specific OS requirements like kernel parameters, in general when an app crashes, you need to look inside the app not outside of it. Furthermore, without a foundation in operating systems, I was unable to explain to them the intricacies of common system functions, much like I'm not qualified to evaluate database architecture decisions.

WOW, were they not happy. I could have absolutely been more helpful had they not been desperately trying to point the finger at my group for the outage. The idea that someone has access and control over their beloved servers has made them resist automation on many servers for months and months.

"Why no you can't just stamp out an oracle server! It is NOT just another app! It requires endless customization and tweaking and DBAs logging in to keep running! Oracle servers are like beloved pets that require care and attention!"

Bullshit. Get with the program. You are not special. Welcome to the cloud, motherfuckers. We shoot servers in the head and bring up identical ones every single day. Oracle is no exception.

Bhodi fucked around with this message at 07:23 on Nov 23, 2013

Bhodi
Dec 9, 2007

Oh, it's just a cat.
Pillbug

dennyk posted:

Also, the DBAs wanting to know what puppet is doing on their system is perfectly reasonable. They don't care how puppet works on a code level, they just want to know what stuff it's managing/changing/adding on their system, which should be easy to figure out and document for them.
For the most part, I completely agree with you. However, the devil is in the details and I cut out a lot of the backstory. Both sides (DBA and Ops) have nearly come to blows over our automation strategy and the whole issue has been building for months. The DBA team's position is "We want to own the DBA servers and we don't even want non-dba teams to have logins; we don't even want you to monitor or touch the OS without first going through a DBA review board. When you bring up a new server, you put a request in and given appropriate lead time, we will handle it." Ops is on the side of "That was fine when we had startup numbers, but that's unsustainable now that we're growing, with with multiple DCs all over the world, thousands of servers and dozens of DBs. We must now bring the DBA servers into line with our overall automated deployment and maintenance strategy, and that includes oracle. Our test deployment of puppet on DB servers in one DC was a success, and so you have no more reason to delay."

When they shifted a fact-finding meeting into an impromptu automation grill session, that was pretty much my limit. I could have handled it better, since honestly the hostility doesn't help the company at all, but without personnel changes or someone above us putting his foot down, nothing's going to be resolved and we'll just continue to go back and forth. Neither my boss or the DBA team is willing to budge, and the capacity problems are still far enough out that management can ignore the issue... for now.

Bhodi
Dec 9, 2007

Oh, it's just a cat.
Pillbug
There are definitely support benefits beyond being able to point a finger at someone.

With Red Hat, For example, we used an extremely long resolv.conf search entry, up to the limit of the RFC (256 chars), during kickstarting through PXE. However RHEL 5.3 (?) didn't support the length. We opened a ticket and they actually patched the issue for the next release and then backported the patch into a custom PXE image and forwarded it for us to use.

I'd have a hell of a time trying to replicate the fix. I could have probably managed it, but it'd have probably taken me days to get it right.

There are downsides, too, though.

Like the way they Red Hat forces servers to connect to RHN (external internet access) for licensing purposes. You can't simply download security patches for servers unless you buy a $XXk "Satellite Server" which will internally mirror the repositories, or "Proxy Server" which is basically a proxy passthrough. There is no quick, convenient rsync solution to provide local, easily updated repo mirrors for your systems like there is with CentOS, Debian, and basically every other linux distro. Unless, of course, you hack together some awful solution like one "patch" server that downloads and installs every single RPM available while using the "save the the RPM" flag, then shoves all those RPMs into some repo. (Don't do this!)

Edit: for the record, Satellite Server AKA spacewalk is a barely-functioning turd of a tomcat application that was created in-house and some bright guy realized they could sell to customers for lots of money because some companies don't actually want every server to be able to have internet connectivity but still want patches and updates. AVOID! I must have submitted a dozen bugs I found in the first 3 months we used it, to which the reply was "This will be fixed in the next version, there is no workaround, sorry". Hate. Hate. Hate.

Bhodi fucked around with this message at 23:26 on Jan 23, 2014

Bhodi
Dec 9, 2007

Oh, it's just a cat.
Pillbug

dogstile posted:

So where's a good start for SQL that isn't loving oracle? I tried oracle before and it was loving horrible.

For a traditional relational DB, considering Oracle owns mysql now, postgres is your most common other option.

Bhodi
Dec 9, 2007

Oh, it's just a cat.
Pillbug
Well, the last time I touched it was almost 3 years ago, at my last job, but I was pretty sure RH disallowed reposync of the updates repo back then due to licensing issues (as our rep explained it to us). We looked REALLY HARD for alternatives.

I'd like to say Sat Server was so bad RH desperately looked for something, anything else. ;)

I haven't used katello yet, but I'm just glad SS is on long term support.

Edit: 3 years, not 2. Yikes. Time is passing, passing...

Edit2: just remembered our issue with reposync at the time, it only allows (allowed?) you to sync repos of the same arch and OS type as the server it's running. Not good for a heterogeneous environment where we literally just wanted a local yum mirror of all OS rpms, and own repos, to be updated with createrepo. Plus random 3rd party stuff like rsyncing vmware tools RPMs.

Bhodi fucked around with this message at 00:57 on Jan 24, 2014

Bhodi
Dec 9, 2007

Oh, it's just a cat.
Pillbug
Funny, because net connect is heads and shoulders above the next best offering (Cisco any connect, I guess), at least viewed from the user side.

Bhodi
Dec 9, 2007

Oh, it's just a cat.
Pillbug
Don't worry, she's just going to be overruled, told her timeline is unacceptable, and to "make it fit, that's your job".

Bhodi
Dec 9, 2007

Oh, it's just a cat.
Pillbug
Devops from the other side is "Encourage dev to release and test a stable product by waking them up at 4am when their lovely application goes down, rather than have them just throw it over the fence to ops with a 'good luck!' every single release"

Also, devops produces people with a basic understanding of the OSI model and what happens outside their little container. You know, useful stuff like cpu/memory usage, network bandwidth, performance, monitoring, all that stuff baked in at the dev level rather than slapped on at the ops level. Maybe even the ability to understand why they don't just slap together 3 outer joins or why they can't just throw more memory at their app.

Edit: Which is awesome. In theory. I've never worked at a place it happens, but I can dream!

Bhodi fucked around with this message at 21:31 on Feb 20, 2014

Bhodi
Dec 9, 2007

Oh, it's just a cat.
Pillbug
MLP, pretty sure.

Adbot
ADBOT LOVES YOU

Bhodi
Dec 9, 2007

Oh, it's just a cat.
Pillbug

Che Delilas posted:

Pretty much. What sticks out in my memory the most is just how FAST it happens. The second you step out the door, BAM, flop sweat over every inch of you, and I'm not exaggerating for effect here - it was literally one second. I'd much rather have the occasional aggressive dryness that we get up here in winters, you can put lotion on if you need to.
This is truth. It's really unbelievable. It's like every pore in your body convulsively opens up all at once.

Friends don't let friends live down south.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply