|
Storage nightmare! Today I poked at our main NAS. Bad idea. We have two most identical, replicated NAS for our main company storage. Each is a Dual Quad with 96x1.5TB drives. They run NexantaStor and Napp-It. For some time my users have been complaining about terrible performance. Turns out the primary box has 24gb of ram for 90TB of space.... 71.5 of that being used.... With twice daily snapshots turned on... Well that makes since. I ordered 96TB of ram for each one today and ZIL+Log SSD's for when I have a chance to rebuild them. Only about 8TB of that DATA needs to be used on a daily basis and due to the performance issues/me wanting a NFS or iSCSI box for my ESXi hosts, I want to look at either A) Building two new SAN's or B) Buying something from NetApp, EMC, ect. What can you folks recommend for the following environment: 1) Two sites, linked via 100Mb MetroE 2) Three ESXi hosts with vCenter (Two at the main site, third at the second office) 3) 20 VM's (Mainly all webservers/code depositories/AD/DNS/WSUS/WDS/ and maybe exchange 2010 for ~60 users in the near future. (3TB of Thick provisioned VM's right now) 4) 8TB and growing of company/user storage If I built them myself, they would be either TrueNAS (we are IX systems largest partner), Nexanta, or OpenIndiana/Napp-IT based. Preferably with SSD's for OS/Log/Cache and 10x 256gb SSD, 20x 3TB 7200rpm SATA. These would end up being $15k/e without software.
|
#
?
Mar 21, 2012 05:43
|
|
|
|
| # ? Apr 19, 2024 00:28 |
|
|
the spyder posted:Storage nightmare! Out of curiosity, what kind of company/size are you at where production is running on custom setups like this? I would love the opportunity to roll some non-standard storage (minus the headaches, it would be fun to learn), but off the shelf systems are solutions which will not get my department fired.
|
|
#
?
Mar 21, 2012 05:54
|
|
|
20 VMs is taking 3 TB of space? Wow. Lowest end of NetApp filers would be my starting point, but you would need someone in there to actually size it properly. The info you've given, while good, isn't really sufficient for sizing a storage system. Also, only the upper end NetApp filers have more RAM than what you've currently got, if your curious.
|
|
#
?
Mar 21, 2012 06:25
|
|
|
Moey posted:Out of curiosity, what kind of company/size are you at where production is running on custom setups like this? I would love the opportunity to roll some non-standard storage (minus the headaches, it would be fun to learn), but off the shelf systems are solutions which will not get my department fired. DOD contractor, we build large custom imaging systems which require a large amount of storage as we add at least 1tb per day. (If I had to keep a copy in house, I would be adding 10-20TB per day, every day of the week.) When we first eval'd equivalent storage from EMC/NetApp/ect, all the vendors came back with costs so high it was impractical. (Like 1/2 the total cost of our product). Right about then https://www.zfsbuild.com was launched, ZFS was taking off, and around 2009 they built a test system. We still have it, a Dual Quad 5405, 16gb ram, and 48 1x 1tb 7.2k Seagates. Every system has been fairly rock solid minus a HDD failure here or there. Our current system is a Dual Hex Core, 16gb ram, and 90x3tb 7.2k hard drives. They work great for the just mass amount of raw storage we need. The processing is done on our 144/288 core clusters. I literally work in a hardware guys porn store. It is rather fun to find brand new 10G Intel nics, X5600 Xeons, cases of hard drives, ect that people have forgot about.
|
|
#
?
Mar 21, 2012 06:51
|
|
|
marketingman posted:20 VMs is taking 3 TB of space? Wow. Move like 6, it has been a long day. All our data is kept on the main NAS, so mainly only OS/program stuff is kept on the host. Did I mention all my storage is local to the hosts right now  . At least it is Raid 5/6.If this is not enough information, what should I be finding out? Better yet, what questions should I be asking vendors? Thanks for the feedback, I get a bit overwhelmed running this place.
|
|
#
?
Mar 21, 2012 06:53
|
|
|
the spyder posted:Move like 6, it has been a long day. All our data is kept on the main NAS, so mainly only OS/program stuff is kept on the host. Did I mention all my storage is local to the hosts right now 3TB, and by extension 6TB, for 20 VMs, in my experience, is HUGE. Especially if I'm understanding correctly that this is only OS drives. By way of comparison, I have 20 VMs in 160GB in the environment that's quickest for me to log into. From memory, I have 150+ VMs in a single 3TB datastore. I have hundreds more examples, literally, but I'm sure you get my point. Now, of course, this is using things like thin provisioning and dedupe. But these are things you should be using also. From your description of only 20 VMs and 60 Exchange users, that screams small requirements to me. However, I don't know what those VMs are actually doing. You say that you go through something like 20TB of data a day, that indicates to me that you have some highly worked VMs? So a good storage consultant will want to know actual IO requirements per server. I would say IOPS but if you can say "xyz server does a constant 50mb/s to the storage during the day" or something similar that would help. You'd also want to have some historical graphs showing your overall data growth, I'd hope that you have something like Cacti that's been keeping track? Considering what your company does, I don't know whether you/whoever controls your budget would feel a real storage vendor would be cost effective for you. Frankly, in my mind, it always is, and people saying otherwise are just being cheap. You could do two shelves of 3TB SATA from NetApp for a usable of 106TB, and a controller of appropriate capacity. This would in most cases be easily expandable as required, shelf by shelf. Further, there is a new NetApp product coming out imminently for long term archival storage requirements that may interest you, though I won't speak to it as it's not public knowledge as yet. If I were in your position, I would locate the most well regarded NetApp partner in your area and have a consultant in for a sizing excercise. It may very well be that your requirements will still put it at the really high price you mentioned before leaving you to continue on your custom made storage, but I just have an inkling that perhaps the previous quotes weren't well done.
|
|
#
?
Mar 21, 2012 07:19
|
|
|
I really should clarify, I have 6TB total on each host, I really doubt more then 500gb is being used. Sorry, it has been an incredibly long day. I had to review a 6 month plan, spending $200k on 18 different projects ranging from replacing my core switch gear to adding a Metro-E line. 20VM's, light use. Highest one would be Exchange. All our data processing is done via our clusters, so no heavy use on the VM server. I believe 1/2 are web/ftp/blog. The rest are imaging/AD/DCHP(eek)/sftp/ random test box number 5. We have cacti installed, no idea if it has been working (I will check tomorrow). What is a budget? Haha, seriously, I have none. I have been here a month and we just are starting to "consider" budgets, but not for another 6 months. I would agree that I would like a real storage vendor, but you nailed my CEO on the head. I will start asking around, monitoring our boxes, and possibly start probing around for a good NetApp rep. Thank you!
|
|
#
?
Mar 21, 2012 07:46
|
|
|
You're situation there is one I'd love to come into with a NetApp build, and so we could talk rotors. It's a shame I don't live/work in the US. Anyway, you really, really, really need to start some statistics gathering immediately. You keep saying these quite high figures but it sounds like you are making guesses/assumptions. You said between 10 and 20TB a day, and you know, that's a big difference between the two - depending on context that's practically a decision point between architectures. You also haven't spoken to business requirements like HA. Can you build a custom made system that has 5,9s? For example, I can pull out the motherboard from one of the controllers in a NetApp and have no user impact. I can add shelves of disks to a running system. I can add those disks into live aggregates during the middle of the day. I can increase the size of a raid group in an aggregate whenever I want. I can SnapMirror all my data to another NetApp filer offsite with one very simple command line. That offsite filer can be any model NetApp filer I want. There's just so much to think about. Also, bear in mind I only speak NetApp. I doubt I'll ever willingly touch EMC or HP again but the point is, I don't know their current stuff. They could be good too. I doubt it, but you can't just rely on the word of someone that sells NetApp, you know?
|
|
#
?
Mar 21, 2012 08:04
|
|
|
You don't buy netapp unless you need the tools, which I doubt he does. I would personally look at Oracle. Same thing you have today, but with HA and support.
|
|
#
?
Mar 21, 2012 12:17
|
|
|
Yeah but then you're giving money to Oracle.
|
|
#
?
Mar 21, 2012 12:23
|
|
|
adorai posted:Take a look at Nimble. Love my nimble.
|
|
#
?
Mar 21, 2012 13:23
|
|
|
the spyder posted:Storage nightmare! quote:The recommended number of disks per group is between 3 and 9. If you have more disks, use multiple groups. Bluecobra fucked around with this message at 15:04 on Mar 21, 2012 |
|
#
?
Mar 21, 2012 14:52
|
|
|
Aniki posted:That's good to know, I have a meeting with NetApp tomorrow and while I like the sound of their equipment, I fear that even their lowest model, the 2040, may be too pricey for our project. I'll look into that SGI unit and see how it compares.
|
|
#
?
Mar 21, 2012 14:53
|
|
|
Bluecobra posted:How is your ZFS storage pool configured? If you have all 96 drives all in one raidz2 group, performance will be terrible since you are writing files across all 96 disks at once. You should have multiple raidz1/raidz2 groups. Per the ZFS Best Practices guide: 6 Disc groups of RaidZ2. 6 hot spares I believe.
|
|
#
?
Mar 21, 2012 18:42
|
|
|
marketingman posted:You're situation there is one I'd love to come into with a NetApp build, and so we could talk rotors. It's a shame I don't live/work in the US. I believe you misunderstand me, I will never see 10-20TB a day- This is what we have a dozen 336TB ZFS boxes, stationed with the systems for. It was a theoretical. I am not looking for a storage solution for our systems or to replace the huge NAS we have. Those work just fine for what they do. I need a new system to store ~8TB and growing of company data + host my VM's. If I had a small 2TB fast pool for my VM's and 20TB of storage for my company data, I would be fine. I do not believe we will outgrow 20TB of company related data, but I could be wrong. I completely agree on the logging/monitoring. I need to get a better grasp on my needs before I start requesting quotes/sizing. If I built something, it would end up looking like this: Single Hex core processor (On hand/discounted) 96GB ram for cache 2x80gb OS drives 2x 160gb Log Drives 2x 512gb Cache drives 10x 512gb Crucial M4 SSD's (On hand/discounted) 20x 3TB 7.2k Seagate (Oh hand/discounted) This would put me right around $15k. What would I be looking at through a vendor for something similar?
|
|
#
?
Mar 21, 2012 18:51
|
|
|
Anyone have some advice for setting up an Active/active NAS? Openfiler to my knowledge only does active/passive. I know Centos supports GFS so doing an active/active would be doable, but any suggestions about other distros or what not to look into for this would be appreciated. I tried asking a few people in my storage/Vmware class most everyone answers with. "Oh we just pay the EMC/netapp people to set it up, we don't really touch that too much"
|
|
#
?
Mar 21, 2012 18:54
|
|
|
Corvettefisher posted:Anyone have some advice for setting up an Active/active NAS? Openfiler to my knowledge only does active/passive. I know Centos supports GFS so doing an active/active would be doable, but any suggestions about other distros or what not to look into for this would be appreciated. You probably get that answer because it's not a common thing to do because the benefits of active/active for NAS aren't that great. Any single export or share is only going to be accessible from a single physical host, just as it would be in an active/passive config. Yes, you can provide a better balance between all exports if you have active active, but if your intent is to run both hosts at above 50% utilization then when you do have a failure event you're going to be hosed. That said, CentOS would be my choice if I was intent on doing something like that.
|
|
#
?
Mar 21, 2012 19:31
|
|
|
NippleFloss posted:You probably get that answer because it's not a common thing to do because the benefits of active/active for NAS aren't that great. Any single export or share is only going to be accessible from a single physical host, just as it would be in an active/passive config. Yes, you can provide a better balance between all exports if you have active active, but if your intent is to run both hosts at above 50% utilization then when you do have a failure event you're going to be hosed. Yeah I know but assuming your SLA's are set properly, active/active can be a good move even in a host failure(other than users noticing it is slower than usual). Running with active/passive right now and just wanting to explore more into storage. I guess I thought most SA's would know about setting from the ground up Active/<passive> or <active> storage. But is seems most people just pay netapp/emc/dell/oracle to do it, which struck me as odd
|
|
#
?
Mar 21, 2012 19:41
|
|
|
Corvettefisher posted:Yeah I know but assuming your SLA's are set properly, active/active can be a good move even in a host failure(other than users noticing it is slower than usual). Running with active/passive right now and just wanting to explore more into storage. http://sources.redhat.com/cluster/doc/nfscookbook.pdf That *should* cover it. You might also have better luck asking in the Linux questions thread. I really didn't run across active/active NAS too much as an SA since usually if the NAS environment had high enough performance demands to need active/active clustering the customer was usually willing to pay a vendor to do it.
|
|
#
?
Mar 21, 2012 20:08
|
|
|
Corvettefisher posted:Anyone have some advice for setting up an Active/active NAS? Openfiler to my knowledge only does active/passive. I know Centos supports GFS so doing an active/active would be doable, but any suggestions about other distros or what not to look into for this would be appreciated. I played with active/active with drbd+GFS2 on RHEL, but even having a 10Gb infiniband cross connect, writes were pretty slow. There was a little bit of a read penalty presumably from the GFS filesystem, but it was negligible. In fact, drbd wasn't able to utilize even 1Gb. Writes for syncs tended to peak around 80MB/s across the infiniband link. I could have lived with having the data with a very slight lag between hosts, but GFS locking just wasn't working out at all. Just reading a file on one host blocked it from being read on the other host. Perhaps with more tuning it could be usable. I didn't have a chance to play with lvm2's version of block syncing, so I don't know if it is better or worse than DRBD. DRBD is kind of a pain with CentOS 6 as it isn't in the CentOS repo. Had to compile the kernel modules from source (very easy to make your own RPMs from the source), and you have to remember to rebuild it every time you update. You'd probably be better off running drbd or lvm sync with corosync or RHEL's cluster manager and running an ext4 filesystem exported via NFS. Corosync will handle the heartbeat and IP handoff between the pair. Note though that in my testing, there is about 30-60 seconds from failover where the filehandles just stick and NFS retries. In my case, Apache tended to hit maxclients during this time as well, so it wasn't nearly as seamless as I'd like.
|
|
#
?
Mar 21, 2012 20:18
|
|
|
NippleFloss posted:http://sources.redhat.com/cluster/doc/nfscookbook.pdf Yeah My RHCSA book actually has a section on this I just saw. They used fedora but I am sure it will work on Centos, and if I have to use fedora oh well they are very similar. optikalus posted:Thank's I have been playing around with corosync on my Openfiler setups, I still don't know why crm_mon --one-shot -V returns a "connection failed", which is very helpful in troubleshooting  . I have gotten A/P to work a couple times but now I seem to be striking out and can't figure out whyI am following this guide almost step by step although I am using a raid array 60GB instead of a 4Gb disk/partition.
|
|
#
?
Mar 21, 2012 20:26
|
|
|
adorai posted:You don't buy netapp unless you need the tools, which I doubt he does. I met with NetApp today and their tools were a huge selling point. My brother has experience working with HP Lefthand units, which he described as a nightmare, and we were really impressed with how easy to use and powerful their management tools are. We also liked that their SnapShots are done at the pointer level, the lefthand unit just does full backups of the file, so they are very quick and don't take up a lot of space. They also have great tools for managing offsite backups, which something that we will need to do in the not too far distant future. Also, the flexibility to manage drives, shares, and even change the RAID configuration on the fly was very impressive. Honestly, at this point even if we weren't thinking of doing virtualization in the near future, we have enough uses for the SAN to justify the cost. What we're currently looking at is a NetApp 2240 configured with either 12 2TB or 12 1TB drives. We want the 2240, since we'll either be adding 10GB Ethernet now or in the near future, and the 2040 doesn't support that. We also want to have spare drives available, since we recently had an issue with a drive in our database server dying and Dell told us it would take them 3-4 months to provide a replacement. I also like that you can switch between RAID 4 and DP on the fly, so we can temporarily change the configuration in the event of an emergency. I like that it has dual controllers, so that we can update or even hotswap one controller and the secondary controller would still keep the NAS up and running. Admittedly, I went into this just thinking that I needed a SAN for running VMs and I needed a central server to store all of our calls. However, once I got my brother involved in the project and we met with them, we came to the realization that this would solve a lot of other longstanding problems that we've had. We're expecting the quote to come un a little under $30 and we may purchase a second unit with one controller in 6 months to a year that would be used as an offsite backup for everything. Though trying to convey the importance of this equipment to our President will be fun, but my brother will be handling that part of the process. Edit: I am still going to look into Nimble and some other options, but it seems like it is going to be hard to beat NetApp's management software. Aniki fucked around with this message at 21:56 on Mar 21, 2012 |
|
#
?
Mar 21, 2012 21:51
|
|
|
NippleFloss posted:You probably get that answer because it's not a common thing to do because the benefits of active/active for NAS aren't that great. Any single export or share is only going to be accessible from a single physical host, just as it would be in an active/passive config. Yes, you can provide a better balance between all exports if you have active active, but if your intent is to run both hosts at above 50% utilization then when you do have a failure event you're going to be hosed. [ASK] Me about working for a webhosting company that ran both datamovers on an EMC box up to 80% utilization... and what happened when one of them failed. Bonus: The plastic pull tabs that allow you to remove the datamover from the chassis were so brittle they snapped: Queue another two hours of downtime while the CE tried to pry it out with a screwdriver. So glad I don't work there anymore.
|
|
#
?
Mar 21, 2012 22:08
|
|
|
Aniki posted:Edit: I am still going to look into Nimble and some other options, but it seems like it is going to be hard to beat NetApp's management software. Buy a nimble, seriously. I've worked with EMC, Dell, and NetApp. They all have their pros and cons, but when it comes to just ease of provisioning/backup managing backup groups, dedupes, disk quota's and initiator groups the nimble is a hell of a lot less annoying to actually use. Also I'm getting some really good dedupe here and the performance has been absolutely wild (I'm doing iSCSI over 10GE)
|
|
#
?
Mar 21, 2012 22:14
|
|
|
Rhymenoserous posted:Buy a nimble, seriously. I've worked with EMC, Dell, and NetApp. They all have their pros and cons, but when it comes to just ease of provisioning/backup managing backup groups, dedupes, disk quota's and initiator groups the nimble is a hell of a lot less annoying to actually use. I'm looking at their white paper right now. I'll contact them later today and at the very least, I can use them to compare to NetApp. The bulk of our data won't be subject to Dedupe, since it will be archived calls, which are already compressed, but I know that becomes a lot more important with getting rid of the overhead from running VMs. Do you have a broad idea of how Nimble compares to NetApp in pricing, is it typically cheaper, about the same, or more expensive? What do you consider to be the cons of NetApp and were those cons before or after they changed their management software in November?
|
|
#
?
Mar 21, 2012 22:19
|
|
|
Aniki posted:I'm looking at their white paper right now. I'll contact them later today and at the very least, I can use them to compare to NetApp. The bulk of our data won't be subject to Dedupe, since it will be archived calls, which are already compressed, but I know that becomes a lot more important with getting rid of the overhead from running VMs. Do you have a broad idea of how Nimble compares to NetApp in pricing, is it typically cheaper, about the same, or more expensive? I predate the change. The last time I worked on a netapp was six years ago give or take, and the last thing I did was trade the thing in for an EMC. The box we had suffered from some pretty serious I/O issues. I'm sure their culture has changed but I had a hell of a time getting support out of those guys. Meanwhile Nimble actually drove an engineer down and we step by step integrated my entire virtual environment into the new storage array. No reading a whitepaper, no reading best practices, boots on the ground giving a hand. For dedupe on virtual machines I'm using about 2TB of *Provisioned* storage for OS VMDK's that is compressing down to about... 90G, give or take. For my stuff that's not easy to dedupe (Fileserv) I'm getting about 1.31X. If you do anything with SQL it dedupes there by a considerable amount as well.
|
|
#
?
Mar 21, 2012 22:31
|
|
|
How much does the Nimble stuff cost? Like the cheapest CS210?
|
|
#
?
Mar 21, 2012 22:54
|
|
|
Rhymenoserous posted:I predate the change. The last time I worked on a netapp was six years ago give or take, and the last thing I did was trade the thing in for an EMC. The box we had suffered from some pretty serious I/O issues. They already mentioned that they would send an engineer out to help with the implementation and we would get 4 hours of training. We do use SQL, we're currently running Oracle (PLSQL), but we may change to another variety of SQL when we launch our new database. That being said, our database doesn't take up a lot of storage space, so I don't know how much overhead it would really create, though the SAN would be setup to run nightly dedupes anyway. In November NetApp switched a graphical interface for managing all of their drives. My understanding is that before that, you had to do everything through the command line, but their software is very straight forward and you can pretty much do everything straight from the GUI. I feel like I have a pretty good handle on how to use it just from the sales presentation. I'll definitely talk to Nimble. I have no brand loyalties or existing NAS architecture to worry about, so I'll see what they have to offer.
|
|
#
?
Mar 21, 2012 22:59
|
|
|
Rhymenoserous posted:I predate the change. The last time I worked on a netapp was six years ago give or take, and the last thing I did was trade the thing in for an EMC. The box we had suffered from some pretty serious I/O issues. Nimble is a start-up. That will ensure a certain level of service to new customers because they desperately need to find them and keep them, but it also means a lack of total resources available. The price you pay for that is they will generally have fewer resources and relationships with other technology vendors so you have fewer assurances that their gear will work harmoniously with Cisco, or Brocade, or whatnot, and the chances of finger-pointing between vendors when something goes wrong is much higher. Also, NetApp management tools have changed completely in six years. Meaning that they actually have tools that are designed to manage an enterprise, and not a single array, though you can do that too. Nimble is comprised of a bunch of ex-NetApp dudes, so I wouldn't be surprised if there are a lot of similarities.
|
|
#
?
Mar 21, 2012 22:59
|
|
|
ok so obviously SAN pricing is always voodoo, but here is my story: I currently have about 5TB of 15kFC and 5TB of SATA disk on a NetApp 3140 HA pair that I replicate to a 2050 with 10TB of SATA. We are considering upgrading the 3140 head to a 3240, buying 2 shelves of SATA, and moving the 3140 head to our colo with 2 additional shelves to make up for the 20 disks we lose from the 2050. This gives us more IOPS at our colo/DR site. Long term, we plan to do this type of activity every 3 years. How much Nimble should I be able to get for the equivalent cost of a 3240 HA pair with complete pack and 2 shelves of SATA?
|
|
#
?
Mar 21, 2012 23:54
|
|
|
I'll hopefully have a price comparison between Nimble and NetApp by the end of the week. Looking at their site, it seems like the CS220G would be their comparable unit to the NetApp 2240. They talked a lot about using SSDs for fast write times, but it kind of reminded me of people using SSD cache drives along with a mechanical drive. I know the two processes aren't the same, but it's the easiest analogy for me to make. It also sounds like they do compressed snapshots instead of the pointer level snapshots NetApp does. They also do inline compression, so the data is being compressed as it's being written to the low speed mechanical drives. On the surface, it sounds like it may be a good fit for us, but I need to look into their management software and how their cost compares to NetApp. I also wonder what sort of redundancy they have in place for the SSDs. Do they use multiple hotswappable drives or is the RAID6 only used for the larger mechanical array? I'm sure it's reliable, but I'll be curious to ask them how it works.
|
|
#
?
Mar 22, 2012 00:40
|
|
|
Aniki posted:I'll hopefully have a price comparison between Nimble and NetApp by the end of the week. Looking at their site, it seems like the CS220G would be their comparable unit to the NetApp 2240. They talked a lot about using SSDs for fast write times, but it kind of reminded me of people using SSD cache drives along with a mechanical drive. I know the two processes aren't the same, but it's the easiest analogy for me to make. It also sounds like they do compressed snapshots instead of the pointer level snapshots NetApp does. They also do inline compression, so the data is being compressed as it's being written to the low speed mechanical drives. On the surface, it sounds like it may be a good fit for us, but I need to look into their management software and how their cost compares to NetApp. There isn't really much information available yet on how Nimble's filesystem works under the covers, but in the basics it doesn't seem very different than other write-anywhere transactional filesystems like ZFS or WAFL. They integrate the SSD layer with the hardware and pretend it's of one piece, but ZFS provides the ability configure SSD as read cache and NetApp sells flashcache that does the same thing. If the management pieces are there then it sounds like it will be a nice option for smaller companies. Currently their lack of any scale-up or scale-out ability will hinder them in medium and large sized companies. They will also have some issues with high throughput workloads that don't fit neatly in cache. The SATA tier might as well be considered near-line backup storage. The spindle count and disk speed is simply too low to do anything else, so if you have a read workload that you cannot ensure remains on the SSD layer you'll have problems with no real recourse to add more IO through expansion. What little technical marketing they have is a bit disingenuous as well: http://www.nimblestorage.com/blog/a-comparison-of-filesystem-architectures/ That states that they somehow magically avoid ever having to do a partial stripe write by running a background scanner to re-arrange blocks to create full stripes that are available for writing. This isn't a magic trick, WAFL has a scanner that does the same thing. ZFS handles things slightly differently by allowing variable stripe width, but the upshot is that all three systems "only" do full stripe writes. In practice, as filesystems become very full, it becomes nearly impossible to always guarantee a full stripe is available on which to write, especially if your system is already under significant load. BTW, in the comments for that link the author notes that they write to SSD in full stripes, so yes, they do Raid-6 there as well. Some other things to note from a comparative perspective are that they do in-line compression, but no de-duplication. On the most recent versions of OnTap you can do both. They also only do iSCSI currently, so if you want CIFS or NFS or FC you're out of luck.
|
|
#
?
Mar 22, 2012 02:27
|
|
|
Here's how I break it down, personally: How responsible do I want to be for this data? If you're OK with being responsible for the SAN and application compatibility and application backups and replication and all of that, then your best money is rolling your own. You can use ZFS or OpenFiler or OpenIndiana or whatever suits you. But at the end of the day, YOU'RE responsible for maintaining spare gear, for ensuring firmware/hardware compatibility, etc. None of the big vendors can compete when hardware is the only line item. NetApp/EMC/etc can't put a full shelf of SSDs and a TB of RAM into a box for less than you can. If you know your poo poo, then you can do this. But you have to have a dedicated SAN person to handle this, because it's not going to be user-friendly. If you have the staff to do this, good luck. If you're OK with not being responsible for your data and you're also OK with the possibility of losing everything because you went with a tier-2 SAN provider, then the newer SAN startups are for you. You might get great support and response time when things are easy, but do they have the depth of experience when things get tough? Or will they sell their business to Dell and let everything go to complete poo poo? You're rolling the dice with an untested/unproven SAN vendor, and you better hope that they keep their promises. If you AREN'T alright with being responsible AND you need your data to be secure, then your tier-1 SAN vendors start making sense. They're more expensive, but you don't have to worry about VMWare or Windows or Citrix or any of your apps not playing nicely. You know backups WILL work, you know replication WILL work, and you're not digging around rsync logs trying to get it working yourself. You're paying for the solution here. I hate to say it, but the "nobody ever got fired for buying IBM" (or Cisco) holds true here. At home, I don't mind spending all weekend loving around with my array. At work, I definitely don't want there to be any fingers pointed at me if something goes up in smoke. I want to know that I made the most conservative possible choices when it comes to a company's lifeblood -- their data. madsushi fucked around with this message at 04:42 on Mar 22, 2012 |
|
#
?
Mar 22, 2012 04:39
|
|
|
I just want to add to the post above me, I've never seen a hugely talented employee in an organisation fired so fast as the time their custom rolled SAN catastrophically failed. Protip: It doesn't matter what C-Level approval you have for the custom SAN project, the C-Level isn't losing their job.
|
|
#
?
Mar 22, 2012 04:50
|
|
|
NippleFloss: NFS is the current best practice recommendation for VMware, so the Nimble units not supporting that is disconcerting. It's not a dealbreaker, but it is certainly something to consider as I move forward. Another thing that they mentioned is that their system is capable of storing 10,000 snap shots per volume and they claim that there are no performance hits for using all of the snap shots. They also claimed that with NetApp that you suffer a performance hit when you use all of your snap shots, though the engineer from NetApp that I talked to stressed to use all 255 snap shots, so I'm not sure who to believe there. They also do their snap shots by compression instead of pointer reference. I like the elegance of the pointer reference, but I imagine that both backup systems would be similary effective. It's good to know that their SSD layer uses RAID 6/RAID DP or whatever proprietary name they call it. With this level of hardware, it's important for us to withstand at least 2 HD failures at any level of the process. madsushi: Getting the SAN is part of a much larger project (replacing the phone system with a software based phone system, virtualizing several servers, and coming up with a real centralized storage and backup system), so we don't have time to mess around with building and maintaining our own SAN. We could do it, but given the importance of our data and the short time frame we are trying to implement everything in, it makes much more sense to have a large company like NetApp available to be responsible for the equipment and guide us through the setup. I'm assuming that Nimble falls into the class of tier-2 SAN providers. They stress their proactive support and seem very eager for business. We don't intend on doing anything too exotic with our storage use, but it is fair to question how their support will function for major problems and what would happen if they were bought by another company. Their systems also have some inherit limitations with how they are built, though I can't see our storage needs ballooning during the lifetime of that SAN, though I do like the scalability that NetApp offers. marketingman: Luckily the company can't function without us, but with this major of a project it makes sense for us to have everything done right. We're used to working with a shoe string budget and makings work, but it's not worth the time or stress in this situation for us to try and cut corners on this project.
|
|
#
?
Mar 22, 2012 05:16
|
|
|
Aniki posted:NippleFloss: NFS is the current best practice recommendation for VMware, so the Nimble units not supporting that is disconcerting. It's not a dealbreaker, but it is certainly something to consider as I move forward. Another thing that they mentioned is that their system is capable of storing 10,000 snap shots per volume and they claim that there are no performance hits for using all of the snap shots. They also claimed that with NetApp that you suffer a performance hit when you use all of your snap shots, though the engineer from NetApp that I talked to stressed to use all 255 snap shots, so I'm not sure who to believe there. They also do their snap shots by compression instead of pointer reference. I like the elegance of the pointer reference, but I imagine that both backup systems would be similary effective. Nimble snapshots are functionally equivalent to NetApp snapshots. They both use the same redirect-on-write mechanism. The only difference is that Nimble does in-line compression of all data, so the snapshots are also compressed. Likewise, it sounds like they actively de-stage snapshot blocks to the SATA tier. But the actual snapshot process is identical, as are the performance implications. So no, there is no performance hit for taking snapshots on NetApp, all the way up to the 255 snapshot limit. NetApp routinely runs their published benchmarks with snapshots enabled because they are essentially %100 free. My guess is that Nimble is claiming that snapshots degrade performance on NetApp because it increases fragmentation within raid stripes and leaves holes. Of course, their filesystem has the exact same problem, which is why they talk about their "sweeping" process that cleans up the holes. NetApp does the same thing with segment cleaning, so that's a bogus claim.
|
|
#
?
Mar 22, 2012 05:55
|
|
|
NippleFloss posted:Nimble snapshots are functionally equivalent to NetApp snapshots. They both use the same redirect-on-write mechanism. The only difference is that Nimble does in-line compression of all data, so the snapshots are also compressed. Likewise, it sounds like they actively de-stage snapshot blocks to the SATA tier. But the actual snapshot process is identical, as are the performance implications. That's good to know, it seemed strange for NetApp to encourage us to use all of our snap shots if there was a performance hit and given the rivalry between those two companies (Nimble was formed by ex-NetApp employees), I'm not too shocked there is some misinformation being passed around. My comfort level is currently higher with NetApp, but hopefully Nimble can give me a better idea of what they have to offer on Friday, though I'll definitely be fact checking their claims.
|
|
#
?
Mar 22, 2012 09:31
|
|
|
Cpt.Wacky posted:How much does the Nimble stuff cost? Like the cheapest CS210? Nimble sales posted:A CS210 is roughly $28,000. For 8 TB raw, 4 TB usable before compression. Is that comparable to other products? Seems like a big chunk of change to me for only 4 TB. I know you're paying for the features and performance, but still.
|
|
#
?
Mar 22, 2012 15:20
|
|
|
Sorry, in the enterprise world that is pretty cheap. I've seen a CS240 with 24TB raw, 32TB usable for under $40K. Now that is serious GB/dollar in addition to IOPS/dollar.
|
|
#
?
Mar 22, 2012 16:07
|
|
|
|
| # ? Apr 19, 2024 00:28 |
|
|
complex posted:Sorry, in the enterprise world that is pretty cheap. The CS240 is 24TB raw and 16TB usable. I believe you're thinking of the CS260, though you have the raw and usable numbers flipped. Those are reasonable prices, but they aren't knocking it out of the park on the GB/dollar axis.
|
|
#
?
Mar 22, 2012 18:03
|
|