Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Serfer
Mar 10, 2003

The piss tape is real





This might not be the right crowd, but any idea how something like a Netapp box has a pool of drives which are connected to the two head units? They can assign which drives go to which in software as well. The only solutions I can come up with involve a single point of failure (eg, using a controller and serving up the drives each as their own LUN to the heads).

Adbot
ADBOT LOVES YOU

Serfer
Mar 10, 2003

The piss tape is real





complex posted:

You're asking how it works? Basically there are two FCAL loops, with a controller at the 'head' of each loop. In a clustered configuration each disk actually has two different addresses, so each head could access it if the other went down. Picture a disk that has two connectors on the back of it, connected to two different controllers. As long as each controller agrees on who is the doing the work, everything is fine cause they won't step on each other.

That's essentially what I figured they were doing. Multipathing, but instead running the second path to the other controller. It seemed to me though, that whatever they were using for that became a new single point of failure.

Serfer
Mar 10, 2003

The piss tape is real





H110Hawk posted:

Via magic, faeries, pixie dust, and most importantly lots of money.
Well, I guess I won't be able to build something similar...

Serfer
Mar 10, 2003

The piss tape is real





Cultural Imperial posted:

If you're asking what I think you're asking, it works like this; with ONTAP 7 and above, the disks contain metadata written at the RAID level which assigns the disks to their respective controllers. This is called software disk ownership.
Well, I was trying to figure out how they had drives connected to both systems really. I wanted to build something similar and use something like Nexenta, but it's looking more like these things are specialized and not available to people who want to roll their own.

Serfer
Mar 10, 2003

The piss tape is real





lilbean posted:

You may be better off looking at just building a couple of systems and using something like DRBD to mirror a slab of disks.
Yeah, that's pretty much where it's going. 100% overhead. But still cheaper that way than buying Netapp or EMC. Which reminds me, despite calling a half dozen times, and having a conference with Sun, they never sent me a quote despite promising that it would get there in x<7 days every time I called.

Serfer
Mar 10, 2003

The piss tape is real





FISHMANPET posted:

I can get a quote from our Sun vendor in a few hours. You've gotta find yourself a local vendor. They're always cheaper than anything we can get directly from Sun or CDWG, although we save money being an EDU.
Well, it's probably too late now. We've already purchased one Netapp for a test setup, and I've got it all running, but we still have a little time before it needs to be rolled out to every other location. So I'll see I guess.

Serfer
Mar 10, 2003

The piss tape is real





EnergizerFellow posted:

With the deduplication need, I'd look into the NetApp FAS2040 (w/ DS4243 tray if you need >12 spindles), which is the only 2000-series worth looking at (otherwise go to the FAS3140, but that's closer to $70K with a DS4243 tray). FAS3140 does have a lot more CPU horsepower and expansion slots, however.

The EMC AX4-5i is down in your price range, but, AFAIK, (still) doesn't have thin provisioning or deduplication.
The low-end NetApp's have dedupe volume maximums. So don't plan on making volumes larger than 1.5tb if you get a 2040.

On the EMC side, look at the NX4, it's basically an AX4 with NFS/CIFS/dedupe. We bought a FAS2020, and the NX4 just seems like the better choice in the long run for us. Fortunately EMC is buying it from us (for more than we paid!). Plus NetApp lied about the storage capacity, failed to mention the dedupe maximums, etc.

Serfer
Mar 10, 2003

The piss tape is real





Oh hi everyone with HP SAN equipment, http://seclists.org/bugtraq/2010/Dec/102

Serfer
Mar 10, 2003

The piss tape is real





Holy poo poo I almost had a heart attack this afternoon. One of our drives failed on a SAN in Oakland, and since my coworker (also in IT) was going over there, I asked him to replace the drive, drive zero (stressing it was drive zero). The drive had failed the day before, and was severely degrading performance. He goes over there, and immediately ejects drive 1. I promptly freak the gently caress out and yell at him for trying to destroy the RAID5 array. He responds "but I thought these drives were hot swappable!"

Thank god I had a hot spare running or bye bye 8tb of data.

Serfer
Mar 10, 2003

The piss tape is real





paperchaseguy posted:

haha, well maybe you should have a support contract and a CE to do disk swaps if your co-workers can't count to zero.

Sad thing is, we do. I can get the part and replace it faster than they can get a CE there. Also we had just moved that office, and for EMC, changing addresses is a multi-day affair, and since I needed the drive replaced as soon as possible, I thought we could do it. I guess I thought wrong.

Serfer
Mar 10, 2003

The piss tape is real





Maneki Neko posted:

Looks like Microsoft released a free iSCSI software target for Windows Server 2008 R2.
Good thing Windows Servers rarely have important security patches that require reboots.

Serfer
Mar 10, 2003

The piss tape is real





Syano posted:

I am pretty curious about this too. Based on a pointer just a few posts up I started researching into Nexentastor and on paper the concept looks fantastic and like a perfect fit for an upcoming project.
The only thing I'd be worried about is that Oracle has killed off OpenSolaris, so who knows who's going to take up the reins on that.

Serfer
Mar 10, 2003

The piss tape is real





ZombieReagan posted:

Definitely, this...FAST-Cache will help keep you from having to add more drives to a pool just for IO most of the time.
Things I hate? EMC's lower end hardware won't support FAST cache. It won't even support SSD's at all. It's loving stupid. We need a SAN at every one of our offices, but can't justify spending $45,000 on a higher end SAN for each location. So we have NX4's currently, and would like to eventually upgrade to VNXe, but without FAST cache, it's still a ridiculous proposition.

I'm actually looking at building our own poo poo with Gluster (I really don't want to do this). It will cost roughly the same, but could be 100% SSD. Why can't someone offer something for smaller offices that still have a really high IO need?

Serfer
Mar 10, 2003

The piss tape is real





Vanilla posted:

EMC is especially anal about flash drives and it's all to do with things like data protection, failure rates, reliability, etc.
I get that, and I think it's dumb. Regardless of the drives costing $10k a piece (which is ridiculous as well), sticking two or four in an NX4 or VNXe is still going to be $30,000 cheaper than going with the next step up that does allow SSD's.

Support for cheaper ones like Intel drives would be nice, but I get their reluctance (all the issues with Intel's firmware), which is why I was looking at building my own.

Serfer
Mar 10, 2003

The piss tape is real





optikalus posted:

Make sure you run some tests before you set your heart on gluster. The performance was just acceptable at best in my test cases, which was still quite a bit slower than even NFS on a LVM vol. Also, this was with a TCPoIB scheme and gluster would hang/crash when using RDMA. My benchmarks were done a year ago, so maybe they're completely invalid now.
I don't have my heart set on anything, I want to explore my options (supporting a bunch of in-house built stuff is really not what I want to do).

Internet Explorer posted:

I read in this thread that you run two, but I am not sure how true that is. Wouldn't surprise me. My point was he said 10k a piece, which is not accurate.
You do need at least two, and I guess I was wrong on the price. I was trying to remember what my EMC rep told me off hand at our last meeting. Either way, I can't use it where I really need it, and since our IO loads are only going to go up, I don't see much future for us in spinning disks, but nobody makes something in SSD that we can afford for our other offices.

Serfer
Mar 10, 2003

The piss tape is real





Things that are pissing me off: EMC.

I don't know what happened in the last six months, but their support has been absolute poo poo recently. Nevermind that the NX4's that we have keep throwing up errors for a successful sector reconstruction, but when they throw out an unrecoverable sector error, EMC requests 8 or so sets of SPCollects (log dumps, drive status, etc), accuses me of creating several of the collects before the error happened (why the hell would I do that?) and then determining that because none of the drives had faulted everything was fine.

After about 18 or 20 of those, they finally get out to replace the drive, and replace it with a bad drive. Try again next day, finally get a working drive, yay. Flash forward a week, and something tries to read off the unreconstructable errors, causing the controller to crash, so the other controller takes over, tries to read the same thing, and crashes. Why didn't they mark these sectors as a part of the replacement? Good question. It caused nearly 18 hours of downtime for one of our offices, and two weeks later, I still can't get them to give me an explanation of why they didn't do anything.

Add that to the fact that they're continuing their "three years and your platform is dead, time to buy again brand new!" system, and the Compellent pitch we just got this week is looking mighty good. Their hardware lasts through revisions, they keep them upgraded, adding new technologies just means adding a card, mixed drive types, tiering, virtualized RAID, they seem to be hitting every sore spot that we're having with EMC. It almost seems too good to be true. Their price is more than we would like, but I'm not dumb, I know we would eventually spend as much on EMC, and then spend more three years later to replace it all.

Does anyone have any experiences that would break the magic spell Compellent has put on me?

Serfer
Mar 10, 2003

The piss tape is real





Vanilla posted:

So speaking quite honestly an NX4 box is basically as low as it gets and is very old. Support is likely the same.

VNX support is much better, VMAX / Symmetrix support is a country mile beyond both of those. The VNX is a world away from the NX4 - the NX4 is from what, 2006?
The NX4 is actually from 2008, we purchased it in 2009. We're having similar support issues on our CX4 as well (which is the same level as the VNX). The real issue isn't that the units are old, the issue is that their support is just terrible. It wouldn't have made a difference if it was a VNX or a VNXe, not replacing a drive the first time it throws an unrecoverable error is just unforgivable. Causing nearly 18 hours of downtime is just the icing on the cake. Add in the other issue we've now run into three times with EMC, the replace entire system every three years problem, and it's just not worth the money to stay with them.

Also, Compellent can do SSD and tiering, I'm not sure where you saw they couldn't.

Serfer
Mar 10, 2003

The piss tape is real





Vanilla posted:

The support you've experienced is truly awful and I'd make sure EMC knew it. Let the rep know, also on any CSAT reports that come through.

Compellent can do SSD drives but the system cache is still stuck on 4GB. I think a medium size array is at about 24GB these days. They can't use the SSD drives as an extension of cache, only as a storage tier.

Not sure if it has improved but last time I was speaking to an admin of Compellent he was bitching that it takes about 5 days to promote data to a higher tier and over to days to demote data to a lower tier.
Hmm, yeah, but using it as tier 0 storage seems like it would be rather effective, even if it can't be used as cache.

Also, just got an alert from the NX4 I was complaining about, MORE UNCORRECTABLE ERRORS. Fuuuuuck.

Serfer
Mar 10, 2003

The piss tape is real





Vanilla posted:

Naaaaaa bro, the VNXe has been in that space for aaages.

Starts at 10k

http://virtualgeek.typepad.com/virtual_geek/2011/01/vnxe-incredible-things-in-tiny-transformer-like-packages.html
I can go on about this for quite a length, but I would highly advise never buying EMC equipment. I'm certainly not doing that ever again. Compellent equipment already on order.

Serfer
Mar 10, 2003

The piss tape is real





Spamtron7000 posted:

Why don't you? If your company is OK with running its infrastructure on whitebox equipment and a heavy software layer then I agree - it's pretty awesome what Compellent has done. Personally, I would only consider them for 2nd tier storage. EMC is a giant pain in the rear end to deal with but in this job market I know I will never lose my job because I selected EMC. They work well. They're reliable as hell and they have a great reputation among the fringe IT crowd (aka, my CTO). If my EMC SAN went down the first thing my CTO would do is call our EMC rep and chew him a new rear end in a top hat. If I had a Compellent SAN as primary storage and it went down, the first thing my CTO would do is say "what the hell is Compellent and who's dumb idea was it to run our infrastructure on it? Fire Spamtron."

It's an exaggeration to show a point. The job market is tough right now. Nobody will second-guess your decision to implement a VNX or Symmetrix. Redundancy is job 1 with EMC. If I was starting from scratch today I would probably go with 3PAR because of how it scales. But since I've got 7 years running EMC Clariions and VMAX without a single second of downtime or performance issues I'm inclined to stay with EMC - the risk of doing anything else is too large.
"Nobody ever got fired for buying IBM"

Compellent is pretty big, and it's owned by Dell now. It's also not "white box" equipment, you're just showing that you have absolutely no idea what you're talking about. It's like saying EMC Avamar is a white box just because it's just a Dell 2950.

EMC support is so horrible it's better to consult the homeless dude begging for change outside of the McDonalds rather than call them. In the past two weeks I've had one SAN go down twice, each time for over 24 hours. EMC's solution is to have a conference call and tell me that not only is it likely to happen again, but the way I should fix it is by buying brand new EMC equipment. Their sales guy had the balls to call me in the middle of the second downtime and congratulate himself for the timing of the new quotes he had made. Their west coast support lead told me that it's likely to happen again, and that simply put, the drives they overcharge us for are worthless and are known problems. They claim that they design for reliability and stability but get a bad sector? Kernel panic, corrupt the filesystem, extended downtime. Not only is this completely unacceptable, they act like it's expected, and not a big deal. Oh yeah, just delete the array, recreate it, rebind the LUN, restore everything from backup. This is not what we purchased a SAN for, this isn't what we pay support contracts for, and isn't sane.

After speaking with a friend who also runs EMC equipment, and having him say that he's had almost the exact same issue on vMax equipment, there's no way I could recommend it.

Additionally, EMC believes that we should replace all our equipment every three years, which is insanity. We can't spend $100,000 every three years, and have to do complete forklift upgrades because they came out with a new model and jacked up support rates to an insane degree. This has hit us twice, and we're not stupid enough to go back for more. Compellent has offered a clearer upgrade path, offers forwards compatibility, and with Dell behind it, I don't see this changing anytime in the future. Additionally, since Dell split with EMC, they are offering steep discounts on replacing EMC equipment, which makes them no longer have the huge price premium over other vendors.

PS, I've literally had to type things for support people because they have no idea where any of the utilities are, how they are used, or even what piece of equipment they should be accessing. They've not known how to open a command prompt, they've referred me to KB articles that simply tell me to escalate the issue to a higher level, and then when I ask to escalate it, they ask if I've followed the KB first. Yeah, I don't know what Dell's Compellent support is like yet, but I'm willing to risk it rather than deal with EMC support again.

Serfer
Mar 10, 2003

The piss tape is real





Spamtron7000 posted:

Sorry - the point of my post wasn't to bash Compellent but if you think Compellent has the same level of redundancy and reliability EMC has, you're kidding yourself. Dealing with EMC is a pain in the rear end but I've had good luck with their support. Their PM's make me want to stab myself in the face and their software is notoriously clunky but I'm still happy with them as my primary storage vendor. I'm glad I haven't had the same experience as you - that's what I was getting at. If I ever do, I'm sure I'd feel the same way.
Dual pathing, dual redundant controllers, dual power supplies everywhere, battery backup on the controller, I'm not sure what kind of redundancy you're implying doesn't exist outside of EMC. Netapp, 3Par, even tiny little Pogo has those.

Serfer
Mar 10, 2003

The piss tape is real





Bluecobra posted:

Here is what a Compellent SC40 controller is:

http://www.supermicro.nl/products/system/3U/6035/SYS-6035B-8R_.cfm

All you need to make it into a Compellent Controller is to put in their PCI-E cards, their fugly bezel, a CPU, processor, and their software. I guess one good point about them is that you can mix and match cards into the same controller since it's just a Supermicro server.
Here's what an EMC Avamar is, http://www.dell.com/us/dfb/p/poweredge-2950/pd by your logic all you need to do to make it into one is their software, and their weird bezel.

I think newer ones use http://www.dell.com/us/enterprise/p/poweredge-r810/pd but my point stands.

It's not that using off the shelf parts makes a whitebox, it's the support behind it (and to some extent, the name) that makes it not a whitebox. If I were to build my own SAN from the same parts, it would definitely be a whitebox, but purchasing it from Compellent, with their software, their support, and their name, makes it more than the sum of its parts.

Serfer
Mar 10, 2003

The piss tape is real





Internet Explorer posted:

So the quality of a SAN depends on how easy it is to rack? Because I am fairly sure our VNX 5300 was a bitch to rack. No sort of "rapid rails" there.
Yeah, our AX4's have two piece rails and were rather a bitch to install. CX3/5 was a total pain too (at least to remove, I wasn't there for installation). And I hate to bring it up again, but the Avamar doesn't use Dell's ready rails either, they use two piece shelf type rails.

Ready rails are pretty awesome though, since I can install them one handed.

Serfer
Mar 10, 2003

The piss tape is real





Hok posted:

You're not the only one, I don't think it's far off either, I'd heard Q4 but I'm not sure if they'll make that now
There was a Dell conference call that required signing an NDA, but I can safely say it's not going to be Q4.

Serfer
Mar 10, 2003

The piss tape is real





Spamtron7000 posted:

I'll get a full writeup from EMC this week.
Good luck with this, the full report I got on my two failures from EMC was "this happens," and "you should buy new equipment to fix it."

Serfer
Mar 10, 2003

The piss tape is real





KS posted:

I've worked with a bunch of vendors and I really like our Compellents, but Netapp and 3PAR are the other vendors I'd recommend without hesitation in the entry enterprise space. I'm sorta surprised to hear you're moving away from Netapp. I doubt Compellent really picks up a lot of their customers.

Be sure you buy series 40 controllers. They will last you a lot longer.

Not the series 50 that is coming out at some point soon? (When does this NDA expire?)

Serfer
Mar 10, 2003

The piss tape is real





For fucks sake EMC.

I've had this one unit showing drive 0 failed for two weeks now. Everyday they call me and ask me to send SPCollects (log files), which I inform them over and over again that I can't do, it errors out. But every day they ask the same thing. They send a tech out, he looks at it, says he can't talk to it, does nothing, and bounces me back to web support, who again does the same song and dance.

EDIT:
What the gently caress? These things run Windows XP Embedded?

Serfer fucked around with this message at 23:20 on Feb 15, 2012

Serfer
Mar 10, 2003

The piss tape is real





Ugh, tonight I was doing some UPS maintenance and had one of the SAN slots down while I was moving in the new UPS, and the other slot decides to panic, taking down the whole thing. Ruined my night. It came back up fine, but more unexpected downtime isn't what I needed.

Serfer
Mar 10, 2003

The piss tape is real





Well poo poo, the same thing that happened to one of our EMC SANs in Los Angeles has happened in our Oakland office now. It hit a bad sector and took down the array. I was up until 2am working with support, but they don't want to do anything because apparently when it hit a bad sector, it also caused the management agents on the storage processors to crash, so they can't see what's going on with the backend. This is insanity, and I'm very glad I had already moved half the stuff to our new Compellent SAN. Unfortunately not the really important stuff yet, but I'm definitely through with EMC forever.

Serfer
Mar 10, 2003

The piss tape is real





Internet Explorer posted:

Well, that's terrifying. What model and OS version? You know, when you're done picking up the pieces of an exploded SAN.

evil_bunnY posted:

There are no words, basically. What model?

It's an NX4 (AX4 with an NFS frontend) version 5.4.something, yes lowend stuff, but it was good enough at the time. Now that we need more, we've ordered Compellent.

Last time this happened they basically said, "lol this happens, buy a VNX to fix the problem!"

See these previous posts:
http://forums.somethingawful.com/showthread.php?goto=post&postid=397522968
http://forums.somethingawful.com/showthread.php?goto=post&postid=398198862

Serfer
Mar 10, 2003

The piss tape is real





Serfer posted:

It's an NX4 (AX4 with an NFS frontend) version 5.4.something, yes lowend stuff, but it was good enough at the time. Now that we need more, we've ordered Compellent.

Last time this happened they basically said, "lol this happens, buy a VNX to fix the problem!"

See these previous posts:
http://forums.somethingawful.com/showthread.php?goto=post&postid=397522968
http://forums.somethingawful.com/showthread.php?goto=post&postid=398198862

So after spending 14 hours Saturday and 10 Sunday in the Oakland office, they say "Oh it's a software problem, you need an upgrade"

"Ok, great, except I requested this upgrade twice before, said I'm perfectly willing to do it myself, but you won't give me access to the files needed to upgrade, and you guys totally ignored my two previous requests."

"But this time we have managers in on the issue"

All you need to have happen if you want something done is to have your SAN totally stop working.

The machine is still throwing unrecoverable parity errors (they say they're false, I doubt it), and storage processor B is crashing every couple hours, but hey, it's accessible, problem solved.

Serfer
Mar 10, 2003

The piss tape is real





Serfer posted:

So after spending 14 hours Saturday and 10 Sunday in the Oakland office, they say "Oh it's a software problem, you need an upgrade"

"Ok, great, except I requested this upgrade twice before, said I'm perfectly willing to do it myself, but you won't give me access to the files needed to upgrade, and you guys totally ignored my two previous requests."

"But this time we have managers in on the issue"

All you need to have happen if you want something done is to have your SAN totally stop working.

The machine is still throwing unrecoverable parity errors (they say they're false, I doubt it), and storage processor B is crashing every couple hours, but hey, it's accessible, problem solved.
Oh hey, it just crashed again and is again offline. gently caress you EMC so hard.

Serfer
Mar 10, 2003

The piss tape is real





FISHMANPET posted:

We're most likely going to be getting Compellent for our main server room on campus, but I'm thinking an Equallogic would be pretty good for another department we support, I guess you'd call it a branch office. Unless Compellent offers something redundant in the 15-20 TB range for same amount of money as Equallogic.
I don't know what kind of education pricing you can get, but the cheapest Compellent will be about $35,000, at least double an Equallogic.

Serfer
Mar 10, 2003

The piss tape is real





As if you needed another reason to not buy EMC, http://www.youtube.com/watch?v=SDhx6ECNww4

Serfer
Mar 10, 2003

The piss tape is real





Speaking of EMC being terrible, I wasted three hours of my time working with yet another know-nothing EMC tech.

I knew things were going to go bad when he tried to cd and ls /dev/hda7. Then he mounted an iso loopback and after doing df, said the upgrade couldn't proceed because one of the drives was 100% full (care to guess which one?).

Serfer
Mar 10, 2003

The piss tape is real





Anyone have any experience in clustered file systems like glusterfs or swift? I had setup a test gluster system like a year or so ago, but it seems like its gotten much better. Swift seems nice because of how it integrates with openstack. I probably wouldn't be doing anything but storing virtual machines on it, so some of the things like remote object access aren't terribly important, but being incredibly scalable with consumer hardware, asynchronous geo-replication, etc, seems like they would be very nice features to have.

Serfer fucked around with this message at 05:58 on Jun 19, 2012

Serfer
Mar 10, 2003

The piss tape is real





Misogynist posted:

A product based on "virtualization bricks" that runs a dead-easy Isilon-like scale-out storage architecture and also hosts VMs would be loving incredible.

Isn't this pretty much what openstack is supposed to be? I mean with an openstack distribution like airframe, you just connect the machine and it boots into openstack, gets its storage added to the pool, and becomes a VM host.

Serfer
Mar 10, 2003

The piss tape is real





MrMoo posted:

That's what pnfs and lustre are for and used in many places exactly so.

It's anyone using ceph? It seems attractive because it combines block and object, but I'm not sure how widely used it is.

Serfer
Mar 10, 2003

The piss tape is real





My boss wants me to build a SoFS system with clustered controllers on an Intel JBOD 2000 system with Samsung SSDs and interposers...

How hosed am I 1-10.

Adbot
ADBOT LOVES YOU

Serfer
Mar 10, 2003

The piss tape is real





Number19 posted:

11

Incidentally that's how many cases of scotch you're going to need to support that thing
Ughhhhh, goody. Hopefully it fucks up during testing so we can ditch the idea.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply