|
Dear IT Goons, After reading the entire SAN/NAS thread here in SH/SC and stumbling on this article: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/ I had a chat with my boss and he’d like me to build out a prototype SAN as part of our push to see if we can start adopting cheaper, commodity or open source devices in non-business critical roles at my company. I have a box running OpenFiler under my desk that I set up with an iSCSI target volume that’s been humming along reliably for a while now. It’s slow, but that’s to be expected of three Caviar Greens in a Raid 5 and communicating with my server over our office LAN. What I’d like to do is build on what I learned making the OpenFIler Box and make a prototype SAN for what may become the bulk storage we deploy to our data centers. The following is what I’d like the SAN to do initially:
The ultimate goal is to build an inexpensive, reliable storage box that can host 20+ TB of data as a slower, cheaper compliment to our real SANs in our corporate and QA environments. I’m looking to deploy these to continue our push towards virtualization and better access to local data. My questions are as follows:
If anyone has any other suggestions or aspects I haven’t thought of, I’d really appreciate it. Thanks in advance!
|
#
¿
Jan 9, 2012 17:18
|
|
|
|
| # ¿ Apr 28, 2024 17:15 |
|
|
madsushi posted:There's nothing wrong with building your own SAN, and it definitely can/does work. Your biggest issue with a homemade solution is going to be support/knowledge. If you build these things yourself, how many people at your organization knows how to fix them? Too often one IT guy gets assigned to "try building a SAN" and then he's the only guy that knows the software and tech well enough to fix/maintain it later. If you're going to roll your own, make sure you keep your coworkers informed and educated so that they can troubleshoot an EMC or a homemade box equally well. Otherwise you end up spending all of your time in the weeds trying to keep these things running. We have a pretty high turnover rate here, so this worries me a little bit. I'm hoping that a combination of using a well documented off-the-shelf product with good documentation on my end should help mitigate support issues if I get hit by a bus, but even then it'll be patchy. Nomex posted:Further to what madsushi wrote, if something catastrophic happens to your home built storage solution, the blame will probably be entirely on you. When you use a major vendor it might not be your neck on the line. My IT Director is immensely knowledgeable, so I assume he wouldn't have sent me off on this project if he wasn't willing to trade some risk for much lower cost per GB. What I'm interested in is what are the best practices for improving the reliability of a SAN beyond the obvious methods (hot spare, server class hardware, backing up critical data off the SAN).
|
|
#
¿
Jan 9, 2012 19:12
|
|
|
Bluecobra posted:Words This was an awesome post; thank you immensely! Bluecobra posted:Supermicro actually does, but for single processors only. The reason for this is that all the Ivy Bridge stuff isn't out yet so you are going to have to wait a few months for the dual processor boards to come out. If you want dual processors now, I suggest getting a 1336-based motherboard instead. Would there be any reason a single SNB CPU wouldn't be fast enough to power a SAN of the type we're discussing?
|
|
#
¿
Jan 13, 2012 21:37
|
|
|
Number19 posted:I'm gonna bump my question about Nimble Storage from a couple of pages ago. They're being pretty aggressive and we're near the point where we'd consider changing directions on storage. They make a good pitch with some pretty nice looking performance numbers and I've read some good stuff on VMware's forums and such but I'd like to know if anyone here has had it running in production and how it's been working out. I'm going to bump this too. The internet seems to have very good things to say about Nimble's product, and the people I know who use them really like them, but it wasn't in a production or similarly stressed environment. ....or do I need to be SA's $250K guinea pig?
|
|
#
¿
Aug 15, 2012 21:27
|
|
|
three posted:How comfortable are you with being one of a very small number of users? It will mean you will be the one running into bugs more often than the other vendors that probably have just as many bugs but have more people to find them and fix them before you notice. Considering the horror stories Goons have shared about their VNX 5300s, I'm not sure Nimble will be any worse. The fact that I haven't found anything really bad about them on the internet actually worries me more than a few 'something broke, support fixed really fast' anecdotes would. Regardless, they're coming in for a Song and Dance (hopefully) next week so I'll see what my upper management thinks about the risk:feature:value ratio they offer. Does anyone here have questions you'd like me to ask their team?
|
|
#
¿
Aug 16, 2012 21:52
|
|
|
oogs posted:Hey! Wooo more Nimble users! I've got a pair of CS460s coming in and I feel like I'm more excited to have them in house then I should be.
|
|
#
¿
Dec 5, 2012 20:47
|
|
|
adorai posted:can someone ballpark what 15-20TB of nimble storage would cost me, per HA cluster? My only requirements are 10GBe, NFS or iSCSI, deduplication, and mirroring. Any specific IOPS requirements? Also, Nimble doesn't do de-dupe, just compression. Not sure if that matters to your usage case (or how much it matters at all), but it bears mentioning. Our CS460s were pushing 6 figures each with 2.4TB of Flash and four years of 4 hour support. If you don't need 60K+ stated IOPS you could get a CS240 for probably well under half what we paid. Feel free to PM me if you have any more questions or if you want me to reach out to my sales rep for sound figures.
|
|
#
¿
Dec 5, 2012 22:45
|
|
|
NippleFloss posted:I don't mean to single you out, this is just a pet peeve of mine with storage talk in general. Talking about IOPs without context is about as useful ask asking someone how fast they can run and them saying "72". You're totally right, I was just crudely try to get at Adorai's workload since it seems to me there are better values for the money than Nimble's SANs if you aren't going to leverage the strengths of their design, one of those being random write performance. I'll phrase my posts better in the future to avoid that kind of unclear language.
|
|
#
¿
Dec 6, 2012 00:05
|
|
|
Gents, I will be racking a Nimble CS460 with a full 2.4TB of cache and wiring it up to our Cisco UCS via 10Gb next week. I know there has been some interest in their products - especially regarding performance, and this is their top end model - so if any of you have questions or test/benchmarks/etc you would like me to run on it, I'd be more than happy to. This pile of parts is my baby, so I have a ton of flexibility as to what I do with it before it goes into production. Feel free to PM me or respond to this post and I'll see what I can do. -B
|
|
#
¿
Feb 15, 2013 23:35
|
|
|
NippleFloss posted:I just don't think Nimble has the resources to focus on flash and their current model is only sustainable as long as SSD density is low and cost is high, both of which are changing quickly. Is there anything specific about Nimble's approach that makes CASL and their filesystem poorly suited to an all-flash architecture, or is it just a matter of focus and resources? I'm curious because Nimble CTO told me that CASL was originally intended to work on PCI-E based SSD installed directly into servers somewhat similarly to how the Fusion IO stack works, but late in the game they decided to build a stand-alone SAN. I asume that's very different than an all-flash array, but I'm just curious about your thoughts.
|
|
#
¿
Mar 12, 2013 19:09
|
|
|
NippleFloss posted:Interesting stuff. I could be wildly off base here, but I still see a future for spinning disks in many storage environments . From my understanding SSDs are amazing for random read writes, (many orders of magnitude faster than spinning disk), decent at sequential IO (2-3 times faster than spinning disk), and pretty terrible at cost:capacity (about 1/10th of spinning disk). Because of the issues with the write endurance that come with shrinking NAND dies, it's not safe to assume we'll continue to see the same linear effect of process advances that we have for the last three decades in the CPU and DRAM markets. Combine that with the uncertain future of our existing CMOS fabrication processes at nodes smaller than 10nm, it's unlikely that NAND will ever come close to the price per capacity of magnetic rotating media. MRAM or some of the ferric memory technologies coming down the pipe might radically change this, but those are too far off to make predictions about. So even if you assume that process shrinks (and other advances in flash based storage) close the effective cost per capacity gap to half of what it is today, you're still looking many times the cost per capacity by going all flash versus a hybrid solution. You know more about common workloads than I do, but I assume the majority of the SAN market lies between super-fast All Flash Arrays like Extreme-IO or Violin memory and the big-boxes-of-SATA disks like DataDomain's products. And it seems that most common use cases are best served either by a couple of products with specific strengths and weaknesses, or a hybrid array that can tier LUNs as most do today. The main difference in our perspectives is that I think NAND won't be able to close the cost gap before we run out of the ability to shrink the chips any more, and that unless something shifts the value proposition harder in favor of all flash for uses in which huge random IO isn't needed, but capacity is, then I think hybrid solutions are the best bet for the foreseeable future. If NAND can close the cost gap to some reasonable level then everything I've surmised will be thrown out the window. And lets face it, if Nimble is still around in 5 years, they'll either have been successful in the market or purchased (likely) by Cisco, so they should be able to fund R&D by then. But that's a big 'if'. 
|
|
#
¿
Mar 14, 2013 19:49
|
|
|
Nomex posted:SSD caching is at best a fad. SSD prices are quickly getting to price parity with spinning disk. In less than 3 years it will probably be just as cheap to outfit an array with solid SSDs and skip the slow tier all together. I can't wait for the day when IO sizing is essentially a thing of the past. The fact that NAND loses both write endurance and speed as process nodes shirk makes me think that we'll never get to the stage where flash storage will be anywhere close to cost competitive with big dumb disks in terms of cost per capacity. And keep in mind we're running close to the end of our ability to continue shrinking transistors even under more ideal applications, so it might take a completely new technology (Phase Change RAM, MRAM, etc) before we finally get to the hallowed land if disk-less storage. I certainly believe NAND will still get a lot cheaper, but we're not going to have 4TB eMLC SSDs selling for $400 in three years like we have SATA disks now.
|
|
#
¿
Apr 19, 2013 00:40
|
|
|
keygen and kel posted:I'm trying to work out a SAN storage agreement with another organization, and i'm not quite sure what the typical usable amount of a SAN is. I've got a Nimble CS460 in production and we see about a ~45% compression rate on our VMware datastores and our nightly snapshots with a 15 day retention takes up about 10% of the total data on disk. Obviously, YMMV, but those are just rough numbers from our environment. Also, with a hot spare that 25 TB is 22 TB.
|
|
#
¿
Jul 19, 2013 23:27
|
|
|
GrandMaster posted:Does anyone know if the VNX2 is released yet or have a release date? We are about to pull the trigger on a smallish VNX5300, but I'd say it's worth waiting a couple of weeks to get the new one.. I just had EMC re-quote me a VNX2 5400 instead of the 5300 because as El_Matarife says, EMC is offering pretty steep discounts on the 5400 in low end configurations until the end of the quarter to avoid bleeding customers until the 5200 comes out. Without any discounts I was told the price on the 5400 was ~15% higher than the 5300.
|
|
#
¿
Sep 11, 2013 22:51
|
|
|
|
| # ¿ Apr 28, 2024 17:15 |
|
|
demonachizer posted:What is the general opinion of Nimble with you guys? We are considering them for a project and like what we see so far but just are wondering about real world experiences also. We had a pair of CS460s backing our production VMware/Database environment at my last job, and I think they're a great solution within their intended market. They're very fast and we had absolutely no issues with our units while I was there. Feel free to PM me if you'd like more details.
|
|
#
¿
Jan 15, 2014 21:51
|
|