Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
devmd01
Mar 7, 2006

Elektronik
Supersonik
Got my first toe in the water today with enterprise storage beyond raid arrays in servers, w000!

Boss: "We had a failed drive on the san, I already swapped in our cold spare, I need you to call it in. Here's how you log in to CommandView EVA, and let me give you a 2 minute tutorial on what buttons NOT to press. Have fun!"

:v:

Adbot
ADBOT LOVES YOU

devmd01
Mar 7, 2006

Elektronik
Supersonik

Internet Explorer posted:

I think the Compellent would be out of price range, if US pricing is similar.

We're currently in the bidding process to replace our entire SAN infrastructure. Talked with my boss a few days ago about it, he's gotten Dell and HP down to 70% off list...even after that, Compellent is still twice as much as the HP for the same basic spec. It sure is nice, though...

devmd01
Mar 7, 2006

Elektronik
Supersonik
Last year, my boss gave me his 2TB external to fill with linux isos to play on the boxee box I convinced him to get.

A couple of weeks ago, he gave me two 1TBs to get him the latest point releases. I kind of split up each folder among each drive as I wasn't sure what he did or did not have.

Today, I see snapshot error messages about backups on one of our command view eva servers, so I log in to take a look.

He had provisioned a 2 TB LUN and was in the process of copying everything from the two 1TBs to it in order to consolidate the folders, then copy it back to the drives.

Gotta love abusing company resources, my boss owns.

devmd01
Mar 7, 2006

Elektronik
Supersonik
What kind of 36gb 10k drive? If it's a raptor, lol.

devmd01
Mar 7, 2006

Elektronik
Supersonik
Boy oh boy, we get to rezone nearly the entire san fabric because we hosed up the zoning assignments when we installed our first blade chassis (all 16 servers can talk to each other...). We're now on our 5th chassis. :stonk:

devmd01
Mar 7, 2006

Elektronik
Supersonik
Brocade 320s, zoning is hosed on both. Typical theoretical setup of dual switches, dual controllers on the compellent, etc.

I know what I need to do to remediate, but it would be easiest if I can build out new aliases/zones on a per server basis instead of server groups and activate those before I deactivate the current ones.

Can I do this? Everything attached is mpio aware.

devmd01
Mar 7, 2006

Elektronik
Supersonik
That's the problem, neither switch is consistent. Ultimately only three servers out of 15 are correctly multi-pathed, the rest are single path due to the hosed up zoning or even fiber cables disconnected. I inherited this mess and I already want to stab anyone who ever touched these. Guess i'm gonna have to do the long haul of documenting everything before I start making changes.

I barely had any fiber switch experience before this and I apparently already know more about how to do things right than the idiots who set it/the servers attached up.

E: yeah, compellent is set up with virtual wwns, at least that part is right.

devmd01 fucked around with this message at 21:31 on Oct 3, 2014

devmd01
Mar 7, 2006

Elektronik
Supersonik
Oh god the change control is in for 1 am Sunday. Babbys first fiber switch zoning teardown and rebuild. Here's hoping I don't take down access to the compellent!

devmd01
Mar 7, 2006

Elektronik
Supersonik

bigmandan posted:

We're pretty comfortable with racking and cabling the equipment, but the setup services include config of the storage units on 4 new hosts we are getting as well. This will be our fist SAN in our environment so having the setup and configuration done for us will be good.

On another note, has any here had experience with Storage Centre Live Volumes? I've read the documentation on it and watch the video Dell put out on it. Seems pretty interesting on paper but I'd like to hear what it's like using it in a production environment.

How much of each tier did you buy? Make sure you have them explain auto-tiering and storage profiles, its pretty straightforward.

devmd01
Mar 7, 2006

Elektronik
Supersonik
For those of you involved in a new SAN evaluation, selection, and migration, how did that process work for you? I'm going to an interview on Monday where my key role in the first year would be doing just that and migrating them away from a 5+ year old EMC, and I want to be able to talk intelligently about going through that process. I have enough hands-on experience with Compellent and fiber channel administration but always on existing systems, or I wasn't involved in the selection/migration process.

First step in my head is data, data, data. How many servers, expected growth, aggregate disk space needed, 95% percentile/average iops per lun, what TYPE of iops/applications using the san, existing storage fabric evaluation end-to-end, as well as 3-5 year business initiatives such as DR or 100% virtualization and expected natural growth.

Next step is compiling it into a usable RFP that we could hand to vendors. Sure, any SAN vendor would be happy to assist you with gathering the data and telling you what you need, but I think it's important to have an idea of what those numbers should look like ahead of time. Once that's compiled, start doing research not only the incumbent's existing offerings, but key competitors. Once you have an idea of what the market looks like, start the vendor contact game and start that process of getting as many free lunches/sports tickets as possible.

Get the proposals, evaluate side by side, narrow to 2-3, ask for demos and start the bidding wars, professional services for the install included of course. Once the SAN is in, hook up a test server and hammer away and play with it for a bit before beginning migration.

Anyone have additional thoughts, ideas on how to go about this?

devmd01
Mar 7, 2006

Elektronik
Supersonik

NippleFloss posted:

right click save as

Perfect, this is exactly what I needed, thanks!

devmd01
Mar 7, 2006

Elektronik
Supersonik
Company I interviewed yesterday talked about buying parts for their 5+ year old emc cx380 on ebay, and the replacement brand new San didn't make the project list for next years budget. Pass. :stonk:

devmd01
Mar 7, 2006

Elektronik
Supersonik
I refuse to support any SAN in a production environment without a maintenance contract in place, unless its something they really, really don't give a poo poo about on there. Ultimately its my rear end on the line if something goes south, and I want that contract in my back pocket to call up and make them fix it when there's a goddamn production outage.

devmd01
Mar 7, 2006

Elektronik
Supersonik

bigmandan posted:

Speaking of Dell storage, we have had our Dell Compellent arrays (sc4020) up and running for about a month now and drat these things are fast. Management so far seems pretty easy and the replication between the two is working quite well. Only issue I really have is that Enterprise Manager is really loving picky about which version of java that should be installed (7u45).

Make sure you set up tiering properly and educate anyone that touches it on how the tiering works. Backup volumes don't need to live in the ssd tier!

devmd01
Mar 7, 2006

Elektronik
Supersonik
Obligatory gently caress IBM post, we have a DS3512 that we need to get firmware to the latest for a vmware upgrade to 5.5, and they lock you out if you don't have a service contract. Guess what these people didnt renew six months ago!

Also, IBM thinks that our units serial is in Singapore. :suicide:

devmd01
Mar 7, 2006

Elektronik
Supersonik
And make sure you're using the correct pathing policies on your data stores per the VMware HCL, bit me in the rear end big time a few weeks ago.

devmd01
Mar 7, 2006

Elektronik
Supersonik
You can't just map the volume to the new server?

devmd01
Mar 7, 2006

Elektronik
Supersonik

Internet Explorer posted:

Ignoring the bigger looming catastrophe, look into SAS DAS units that can allow multiple hosts to connect. Like the Dell PowerVault MD series (non-iSCSI). They are basically OEM hardware and most major storage companies sell an equivalent.

You can get them with dual controllers and dual power supplied and will let you sidestep the switching side of things.

Ding ding. This is the only realistic solution for a two-host VMware cluster that won't be expanded.

Just make drat sure you follow the VMware hcl for round-robin pathing policies, I had a nightmare of a VMware host upgrade a few months back at our sister company.

devmd01
Mar 7, 2006

Elektronik
Supersonik
Can someone point me in the right direction of some decent Netapp 101 resources? All of my storage experience has been with Compellent and a little bit of EMC; and I just threw my hat in the ring for a position that uses UCS/Netapp for their vmware clusters.

I'd like to be able to talk somewhat intelligently about it should I get called. "I haven't used netapp, but <compellent concept> translates to <netapp concept>, etc"

devmd01
Mar 7, 2006

Elektronik
Supersonik

Vulture Culture posted:

Can't answer this without the I/O requirements or some basic idea thereof. How many cameras writing concurrently? What framerate, resolution, codec? Do they stream video directly to the disk or do they buffer files locally and burst-dump the whole thing at once?

Yeah I was about to say, an installation that large your camera installer/var should be doing the legwork for you, and be able to at least provide you with iops requirements, etc.. If they're not going to/can't provide you with a turn-key system or solid specifications for the archiving portion of the storage, then get another VAR.

devmd01
Mar 7, 2006

Elektronik
Supersonik
All of our new Sans for the new colo and dr and accompanying switches have shipped, as soon as we have network up at the end of next week in the colo it's go time. :dance:

devmd01
Mar 7, 2006

Elektronik
Supersonik
edit: nvm

devmd01
Mar 7, 2006

Elektronik
Supersonik
What does everyone use for benchmarking their SAN performance, and what constitutes a valid test? I'm getting a new SC8000-series Compellent installed next week, along with new 10Gig switches. I'd like to validate performance while the engineer is engaged and on-site, so we can tune if necessary. Anything that touches the compellent storage is 100% vmware, redundant 10Gig all the way through in round-robin.


E: Perfect, thanks!

devmd01 fucked around with this message at 15:25 on Aug 24, 2016

devmd01
Mar 7, 2006

Elektronik
Supersonik
I just...I just give up. I've been working through a new datacenter deployment/migration, and we are all ready to go for the migration phase except for the new SAN install.

We got our new SC8000 pair installed in the colo this week, and our last step was to configure replication from our current production SC030 compellent to the new one for the migration process. We also purchased another pair of SC8000s at the same time for DR with a couple of disk shelves.

The SC30 SCOS is too old and the SC8000 SCOS is too new; there's a SCSI I/O card driver mismatch which breaks replication, which we discovered after an hour of troubleshooting and finally calling Copilot support - we had the answer in 10 minutes.

If we backrev the SC8000s enough for the driver issue, we would lose access to the SSD tier until we are completely migrated and can upgrade the controllers. Unacceptable, as I have a really dumb ERP sql system that I need to throw more IOPS at as soon as I can migrate it. We can't upgrade the SC30 to a high enough rev because they are 32-Bit only, latest versions of SCOS are 64-bit. :negative:

So my deployment consultant believes that we can set up the new SC8000s for DR in the colo (which we would need to do anyways for initial replication seeding), and set up a two-step replication process, from our current prod, to new DR SC8000s over the 100mbit mpls, to direct to the new production SC8000s over 10GBE. However, we are missing a set of compellent 10gbe cards, because we were going to take the existing ones in current production down to the new DR, after we did the datacenter migration and had it decomissioned.

I talked to our dell rep to see what options we have, I was hinting pretty strongly that he should find me some loaners for a few months. If not, get me a god damned quote ASAP so I can call my boss on her day off to approve it.

devmd01
Mar 7, 2006

Elektronik
Supersonik
Everything was vetted and we were good for install, the problem is because of the drive mismatch dell introduced with the latest SCOS. There isn't even a bulletin out there, but once we described to the scenario to copilot support he instantly knew what the problem was. That was our last step to get it 100%.

devmd01
Mar 7, 2006

Elektronik
Supersonik

Moey posted:

Yeah, I'll let the thread know if everything goes to poo poo. Spoke with our reps recently and they claimed "nothing will change."

If it goes to hell, I wouldn't mind trying out Pure.

New place has a pure in prod, just set up another on DR, it took us 5 minutes to set up replication today, 2 of that was plugging in the cables.

devmd01
Mar 7, 2006

Elektronik
Supersonik
I will never run a SAN in production without a maintenance contract, period. If the company is willing to accept that risk, then it's time to find a new company.

At the job I just left, I migrated the data center to a colo space on a new SAN, but there was a 3 month period where the old SAN was out support and still running business critical applications, due to various delays. I still puckered anytime a disk failed in the old SAN, even though I had spare drives sitting on the shelf ready to go.

devmd01
Mar 7, 2006

Elektronik
Supersonik
On more goddamn vm to migrate off the CX4 this weekend and we’re kicking that fucker to the curb.

The jr admin wants to take it home, lmao. Have fun getting a 30A circuit installed and your electricity bill doubling!

devmd01
Mar 7, 2006

Elektronik
Supersonik
You’ll need to break redundancy, move one set of cables to the new SAN, sVmotion to the new datastores, and re-establish multipathing once everything is migrated. Hope you can find some easy maintenance windows!

devmd01
Mar 7, 2006

Elektronik
Supersonik
Oh hell yes, we just got approval from the CIO to dump our VNX5400 entirely and upgrade+expand our existing Pure storage. We’ll be 100% flash ssd/nvm across both of our datacenters soon and with Pure on both sides we can start looking at active failover between sites.

devmd01
Mar 7, 2006

Elektronik
Supersonik
Lots of reasons for #3. Separation of I/O workloads, if you have auto-tiering you can pin one LUN to the top tier. Maybe you replicate a couple of luns over to your DR SAN but not everything, keeping a/b pairs of vms separate, etc.

devmd01
Mar 7, 2006

Elektronik
Supersonik
Goodbye VNX5400! We are running the disk wipe and then it is getting yanked out of the production datacenter ASAP.

devmd01
Mar 7, 2006

Elektronik
Supersonik
Yeah but EMC.

I’ve been very very happy with our Pures, they have been rock solid from day one and the support is fantastic. I don’t even bother doing software upgrades, I just put in a ticket with support to upgrade to x version at y time and they do it.

devmd01
Mar 7, 2006

Elektronik
Supersonik
Just did controller hardware upgrades on both of our Pure installs, a+ would have them replace production hardware in the middle of the day again. Zero downtime, zero issues.

devmd01
Mar 7, 2006

Elektronik
Supersonik
VNX5300 is wiped, powered down, and cables ripped out. We’re gonna have a company come pick it up next week and give us $300. Good riddance! We are now 100% Pure only.

devmd01
Mar 7, 2006

Elektronik
Supersonik
There were other reasons, but many years ago I straight up told a company why I refused their job offer without even seeing numbers - they were using a used CX4 without any kind of support contract and were buying parts off of eBay. gently caress. No.

devmd01
Mar 7, 2006

Elektronik
Supersonik

Pile Of Garbage posted:

That's why you, or rather your employer/customer, pays for enterprise kit with support so that the liability is shifted up to the vendor.

This right here. Like hell if I’m gonna deal with incredibly proprietary technology that is the core of our infrastructure without someone to point the finger or have a ticket in with if something is wrong.

devmd01
Mar 7, 2006

Elektronik
Supersonik

Bob Morales posted:

Really liked Veeam in the past. We are using Druva at the new place. It's pretty dumbed-down but it works.

Same. Thankfully I’m not the backup admin bitch so I literally have no idea how it works except for how to do a VM restore if needed.

Works great for O365 mailboxes though! I don’t think we have really advertised the capability to people even though the tile is available in Okta for people to use.

devmd01
Mar 7, 2006

Elektronik
Supersonik
Pure is the best, we haven’t had a single drat issue with them. The middle of the day controller upgrades with zero downtime is pretty cool too.

Adbot
ADBOT LOVES YOU

devmd01
Mar 7, 2006

Elektronik
Supersonik

Internet Explorer posted:

We're using FC, so I can't speak to that side of things, but overall I have been quite happy with our Pure arrays. My only real gripe is that you can't do updates yourself, which seems weird to me in 2022.

Hell no. I don’t manage our pures anymore, but when I did it pretty much was “call support, schedule the upgrade, put in CR, enable support access in the right window” and sleep like a baby.

When we did our evergreen controller upgrades, it was middle of the day Monday with zero issues.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply