|
Misogynist posted:I just moved part of my ESXi development environments off of local storage and onto a 48TB Sun x4500 I had lying around, shared via ZFS+NFS on OpenSolaris 2009.06 over a 10GbE link. What zpool configuration are you using? Any specific kernel tweaks?
|
#
?
Aug 6, 2009 20:29
|
|
|
|
| # ? Apr 23, 2024 20:55 |
|
|
Bluecobra posted:Are you using these 10GbE cards? They have been working extremely well for us but on some servers we added another card and turned on link aggregation. There is also this new feature in Solaris 10 Update 7 that may or may not be in OpenSolaris: I also don't think our non-HPCC switching could handle it well. We're running some impressive Force10 gear on the HPCC side, but I think we're bottlenecked down to 10gig to any given rack group for our regular network. H110Hawk posted:What zpool configuration are you using? Any specific kernel tweaks?
|
|
#
?
Aug 6, 2009 21:52
|
|
|
Pretty neat deal http://www.xiotech.com/Products-and-Services_Cash-for-Disk-Clunkers.aspx?elq=ea858ca57c4f46c3a1cd0dd94711be7d
|
|
#
?
Aug 20, 2009 20:54
|
|
|
Misogynist posted:If I can find some sane way to replicate this to another Thumper, I will be a very, very happy man. I talked to a guy at a USENIX conference last year whose company was using dozens of X4500s as the storage backend for one of the biggest ESX deployments I've ever heard of (300k+ IOPS). At the time I was dealing with HP SAN issues and was very jealous. Now not so much, but NFS definitely seems to scale better than fiber channel for big clusters. 4540s are probably the best storage possible for ESX.
|
|
#
?
Aug 21, 2009 03:18
|
|
|
This is probably an invitation to all sorts of weird PMs, but do any of you do SAN consulting and/or sales? I'm pretty sure I'm in over my head with my boss's request to virtualize several of our bigger physical servers. The virtualizing part is easy enough, but I don't really know enough about how to measure my hardware requirements, or how to shop intelligently for a low-end SAN that will meet those requirements, and I don't want to clutter the thread with all my newbie-level questions.
|
|
#
?
Aug 24, 2009 15:11
|
|
|
Weird Uncle Dave posted:This is probably an invitation to all sorts of weird PMs, but do any of you do SAN consulting and/or sales? I work for EMC, there's at least one other person here who does, and several other professionals. You really won't clutter up the thread, it's not super active. Fire away with any questions.
|
|
#
?
Aug 24, 2009 16:02
|
|
|
Weird Uncle Dave posted:This is probably an invitation to all sorts of weird PMs, but do any of you do SAN consulting and/or sales? Profiling Windows and the various MS apps is pretty easy -- your best bet is to download the 180-day trial of Systems Center Operations Manager, load it onto a VM, point it at your AD/Exchange/SQL/whatever servers and let it go to town. Within a couple of days you should have most of the relevant performance information you need. Microsoft also has a lot of best practices guides for how to obtain relevant performance information out of applications like Exchange, just Google for the whitepapers. Linux is a lot trickier, especially if you're running on RHEL/CentOS or something else that doesn't have iotop and the other nice stuff that have made it into the system in the last couple of years. You'll have to babysit iostat for awhile.
|
|
#
?
Aug 24, 2009 16:10
|
|
|
We're so small, we don't even have a Windows domain. The Windows servers to be virtualized (all three of them) are standalone. Is Windows perfmon, monitoring number of disk operations per second, a decent approximation of IOPS? I'm only really worried about disk speed for one server (about 4000 email addresses); the others (a little-used Web server and a small database server) aren't a big problem. Network bandwidth and RAM are cheap by comparison. Most of our Linux servers (yes, they're pretty much all CentOS) have already been virtualized, so I should be able to get the data I'll need from VMware ESXi itself. Right now, there's about twenty guests on one host's local storage.
|
|
#
?
Aug 24, 2009 18:02
|
|
|
Weird Uncle Dave posted:Is Windows perfmon, monitoring number of disk operations per second, a decent approximation of IOPS? I'm only really worried about disk speed for one server (about 4000 email addresses); the others (a little-used Web server and a small database server) aren't a big problem. Network bandwidth and RAM are cheap by comparison. Exchange 2007 average user profile estimates .4 IOPS per user account. I like to use .5 IOPS because that gives some wiggle room and is easy to calculate: 2000 IOPS. A 15k FC drive maxes out on typical IO sizes at 180 IOPS. You could probably do all this on one 15 disk enclosure of FC drives, maybe a little more depending on the size. perfmon would give you a realistic estimate of IOPS.
|
|
#
?
Aug 24, 2009 20:01
|
|
|
That sounds like way more than what I'd need. Right now, the mail server is an eight-year-old Dell PowerEdge, and all the email is on three 15krpm SCSI drives, RAID-5'd, and everything works perfectly well. Left perfmon running over lunch, and the drive with all the mailboxes peaked at about 100 IOPS. Granted, that was lunch, and I'll leave things running for a couple days and peek in on them occasionally. Most of the other servers are very light on disk use (static Web pages, DNS, very small databases, things that are more memory-bound than anything, fortunately RAM is cheap). So I should be okay in terms of disk speed; looks like even the cheapest EqualLogic iSCSI SAN will probably have plenty of that, both for now and for later. (I'm not wedded to EqualLogic by any means, just that they're Dell, and we've generally had good luck with Dell in the past.) Assuming we go iSCSI (FC is probably way out of my budget), do I need any special magic gigabit Ethernet switch, or will any decent one do?
|
|
#
?
Aug 24, 2009 20:18
|
|
|
Weird Uncle Dave posted:That sounds like way more than what I'd need. Right now, the mail server is an eight-year-old Dell PowerEdge, and all the email is on three 15krpm SCSI drives, RAID-5'd, and everything works perfectly well. A decent one will do. I'm partial to the HP Procurve line, although if you want jumbo frames + flow control you need a higher end switch. The 2824 would probably suit you fine (I have 2848s and they work flawlessly). It might be possible to go with a lower cost, unmanaged Procuve, but I don't know what would be capable off the top of my head.
|
|
#
?
Aug 24, 2009 21:22
|
|
|
Weird Uncle Dave posted:That sounds like way more than what I'd need. Right now, the mail server is an eight-year-old Dell PowerEdge, and all the email is on three 15krpm SCSI drives, RAID-5'd, and everything works perfectly well. If you have 4000 addresses, but many aliases, unused accounts, etc, that will obviously lower your requirements. Maybe a 4+1 SATA appliance would be better for you. SATA does 50-80 IOPS as a rule of thumb.
|
|
#
?
Aug 24, 2009 22:40
|
|
|
paperchaseguy posted:I work for EMC, there's at least one other person here who does, and several other professionals. You really won't clutter up the thread, it's not super active. Fire away with any questions. Ahhh another EMC brother  and joined three days before me! Weird.Weird Uncle Dave - on the MS site there's a tool that will give you their recommendations for disk based on users, workload, etc. However if it's running fine on three disks and everyone's happy then three disks it is. Leave perfmon running for longer - 9AM is going to be the key time for things like Exchange. If you can also use perfmon to get items such as the read / write ratio of your exchange environment that would help translate the IOPS that is actually hitting the backend (writes have an overhead with R5 due to the parity calc and additional IO required).
|
|
#
?
Aug 24, 2009 23:33
|
|
|
We're not even running Exchange. It's a weird old mail package (Ipswitch Imail), that stores users' mail in per-user files (Unix .mbx style, as opposed to "one file per piece of email" or "Exchange-style giant database for everyone"). Perfmon will be running for a few days at least, both here and on the other servers I'm looking to consolidate (though they basically don't count - our billing database averages about 10 IOPS, and no that's not a typo).
|
|
#
?
Aug 25, 2009 01:02
|
|
|
Haha, oh man, I pity you remaining IMail admins. 
|
|
#
?
Aug 25, 2009 16:12
|
|
|
We are currently looking at replacing our aging ibm san with something new. The top two on our list are a pair of Netapp 3020s and a 2050 for our offsite or a pair of EMC Clarions. I am interested in looking at a dual head Sun Unified Storage 7310 and a lower end sun at our offsite. The numbers seem to be literally half for the Sun solution, so I feel like I have to missing something on it. For usage, the primary purpose will be backend storage for about 100 VMs, some cifs, and some iSCSI/fibre storage for a few database servers. Any thoughts from you guys?
|
|
#
?
Aug 26, 2009 00:24
|
|
|
adorai posted:I am interested in looking at a dual head Sun Unified Storage 7310 and a lower end sun at our offsite. The numbers seem to be literally half for the Sun solution, so I feel like I have to missing something on it. We use the Sun Storage 7410 where I work. We bought it back in December when they didn't have the 7310 yet. We bought just the single head system. We use it for NFS/CIFS and iSCSI. It does the job and fit our price point (non-profit). My feelings are we purchased it 6 months too early. The Sun Storage line only hit the market back in November I think, so it is still fairly new technology. I don't remember them supporting fibre SAN... I remember promises about it being worked on, so if they support it now then it's relatively new. I don't know if Sun still has it, but there was a try before you buy program. You had to be using one of their pre-built configurations approved for the program though. If you go the Sun route, I would suggest against OS upgrades until at least a week has passed from the release date. I would also suggest you never upgrade to a base quarterly release. We were screwed hardcore with kernel panicks on 2009.Q2.0.0 (fixed the slow MS Office document issue on CIFS that management was so adament got resolved ASAP though). I would also recommend the dual-head clusterred version for this reason. The Fishworks interface is really cool though. Before the Sun Storage I never knew how many IOPs we needed :-). The fishworks people keep a wiki of release notes here: http://wikis.sun.com/display/FishWorks/Sun+Storage+7000+Series+Software+Updates Edit: forgot to mention it's much faster than our antiquated 4 year old NetApp FAS270 was. HorusTheAvenger fucked around with this message at 01:24 on Aug 26, 2009 |
|
#
?
Aug 26, 2009 01:19
|
|
|
Misogynist posted:Haha, oh man, I pity you remaining IMail admins. Hey, there's nothing wrong with Imail. Well, that's a lie, there are lots of little things wrong with Imail, but we've gotten used to them. Stockholm syndrome. Anyway! Now that I have a better idea of what I need, what's the best way to go about buying it? I'm pretty sure we're comfortably on the low end - Dell EqualLogic PS4000, Dot Hill 2330, something like that. Nobody has pricing listed publicly on their Web sites, of course, and I still don't know all that much about the intricacies of these products (or how they'll work in the real world). How do I make sure I'm getting something I need but I'm not getting completely ripped off? Chances are, whatever I buy will cost more than my car, which to date is the largest purchase I've ever made (and my car was bought with my own money, whereas this is the boss' checkbook, so I figure I should be even more prudent). If any of you sell something suitably low-end (just a couple terabytes, iSCSI, probably 800-1000 IOPS), feel free to PM me and we can talk.
|
|
#
?
Aug 26, 2009 16:45
|
|
|
adorai posted:We are currently looking at replacing our aging ibm san with something new. The top two on our list are a pair of Netapp 3020s and a 2050 for our offsite or a pair of EMC Clarions. I am interested in looking at a dual head Sun Unified Storage 7310 and a lower end sun at our offsite. The numbers seem to be literally half for the Sun solution, so I feel like I have to missing something on it. Is your offsite storage intended to be a replication target? What applications are you looking at running on the storage? MSSQL? Exchange, etc? A NetApp box will do CIFS, NFS, iSCSI, and FCP all in one unit. I think the EMC Clariion only does iSCSI/FCP out of the box but you can throw a celerra in front of it or build out a fileserver. What licensed options did you include with your storage from the other two vendors?
|
|
#
?
Aug 26, 2009 18:11
|
|
|
adorai posted:We are currently looking at replacing our aging ibm san with something new. The top two on our list are a pair of Netapp 3020s and a 2050 for our offsite or a pair of EMC Clarions. I am interested in looking at a dual head Sun Unified Storage 7310 and a lower end sun at our offsite. The numbers seem to be literally half for the Sun solution, so I feel like I have to missing something on it. Personally I say avoid SUN. Can't go wrong with Netapp or EMC, both good kit. EMC Clariion will do as 1000101 says but EMC has a Unified array that will do the whole shebang - FC, iSCSI, CIFS, NFS.
|
|
#
?
Aug 26, 2009 22:57
|
|
|
What are your issues with the Sun kit?
|
|
#
?
Aug 27, 2009 05:29
|
|
|
TobyObi posted:What are your issues with the Sun kit? Does Sun storage have any VSS snapshot software for Windows now? Back in December they did not and neither did they have any vmware compatible snapshotting or anything else. I really liked their hardware, but their software was just way too raw for my liking. I went with Equallogic instead for that particular project and we got bunch of NetApp boxes since for a few other projects. However, if Sun software stack improved (or you don't need it and NFS/iSCSI on Linux/Solaris is good enough), I'd definitely take a look. I did hear bad things about support, and heck, even their pre-sales tech support was meh.
|
|
#
?
Aug 27, 2009 05:52
|
|
|
oblomov posted:Does Sun storage have any VSS snapshot software for Windows now? Back in December they did not and neither did they have any vmware compatible snapshotting or anything else. I really liked their hardware, but their software was just way too raw for my liking. They have supported Shadow Copy on CIFS shares since the 2009.Q2 release: http://wikis.sun.com/display/FishWorks/ak-2009.04.10.0.0+Release+Notes quote:Microsoft shadow copy for shared folders No tie-ins like SnapManager for Exchange like what NetApp has for snapshotting iSCSI luns yet afaict though.
|
|
#
?
Aug 27, 2009 13:18
|
|
|
TobyObi posted:What are your issues with the Sun kit?
|
|
#
?
Aug 27, 2009 14:59
|
|
|
Bluecobra posted:Probably because he works for EMC? Yes, but if that was the only reason why would I recommend Netapp in the same breath? Do we wub them <3 <3?  TobyObi posted:What are your issues with the Sun kit? So my issues with the SUN kit....  Firstly there’s a lot missing in the SUN range that people have come to expect and that the EMC and Netapp boxes are offering. Replication – Not too sure about Netapp but EMC replicates based on RPO. Got a file system that’s pretty important? Set a 1 minute RPO. It’ll even tell you if it can’t maintain this. With the Sun box it’s the most basic replication ever. No throttling or any form of control – I think it’s free. It’s also only Asynchronous, no sync. Then there are a few things that the Netapp and EMC arrays do have – such as deduplication (at no extra charge on either), the ability to have a wide range of drive sizes and type (not just 1TB drives or a very limited selection), Fibre Channel, RAID 1, RAID 10, RAID 3. I could go on forever. No kind off application snapshot integration such as with NTAP Snapmanger or EMC RM. Performance monitoring that only goes from CPu to User (what about the actual backend?? The file system isnt everything). They’ll have a new range out eventually. Naturally they’ll have Fibre Channel by then and you’ll probably find the usual story – no upgrade path, gotta buy new. Then onto Sun as a business. Never have I seen a company have so many fantastic ideas and fail to bring them to the market. They bring these items out to market, persuade a few thinkers to adopt these revolutionary products and then drop them. Leaving their key sponsors holding a product that is end of life. People just have no confidence in them. Vanilla fucked around with this message at 17:40 on Aug 27, 2009 |
|
#
?
Aug 27, 2009 17:32
|
|
|
To clarify some things on the NetApp: It supports RAID 4 and RAID 6 (NetApp calls it RAID DP). It in fact does support replication based on your RPO or it can do synchronous. One thing I found with some Sun Storage is that it doesn't appear to do active/active very well. Which is to say if I have a LUN presented off controller A and somebody tries to access that same LUN via controller B, B ends up usurping the LUN. I had a customer who build their SAN design where controller A was entirely on fabric A, and controller B was entirely on fabric B. When one of his ESX hosts had a path failure the LUNs kept hopping back and forth between controllers which drug performance down. Not sure if this is still an issue (it's simple enough to fix) so if someone has a similar setup and could confirm that would be great.
|
|
#
?
Aug 27, 2009 18:04
|
|
|
Vanilla posted:the ability to have a wide range of drive sizes and type (not just 1TB drives or a very limited selection), Fibre Channel, RAID 1, RAID 10, RAID 3. I mean, I have an EMC AX150i. I haven't written off the entirety of EMC, just the shitheel who thought that he did a good job whipping that up in about 20 minutes. I'm not disagreeing on some of your other points though.
|
|
#
?
Aug 27, 2009 22:27
|
|
|
TobyObi posted:To be honest, this sounds more like a gripe with a certain model, rather than an entire company. I have a pair of Sun arrays, that are "entry level" that support all of these things. Indeed it is a gripe - this is supposed to be their super product to destroy EMC and Netapp but is so far off the mark and missing so many key, simple things that it's laughable! It's the typical SUN home grown build hardware around ZFS. It's so good it'll sell itself!  quote:I mean, I have an EMC AX150i. I haven't written off the entirety of EMC, just the shitheel who thought that he did a good job whipping that up in about 20 minutes. 35 minutes*. *included 7 minutes cigarette break.
|
|
#
?
Aug 27, 2009 23:08
|
|
|
A while back I mentioned filling up a 4540 with lovely consumer disks, and I went ahead last Friday and did just that. Well, not full yet, but I've replaced to begin with one ZFS vdev of 6 disks plus one of the available hot spares with Seagate 1TB 7200.12 drives (100CDN each). In the last weekend and week we've pounded the poo poo out of them since our 4540 is our primary backup server and acts as a disk-to-disk staging area. No problems whatsoever so far. Also, I think my manager is ready to blow me for being able to expand at 20 cents per usable gigabyte (after RAIDZ2 and a spare are taken into account).
|
|
#
?
Aug 28, 2009 21:09
|
|
|
Has anyone heard of backblaze? Seems like a personal backup service, but their blog post on their infrastructure is pretty cool - http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
|
|
#
?
Sep 2, 2009 18:49
|
|
|
da sponge posted:Has anyone heard of backblaze? Seems like a personal backup service, but their blog post on their infrastructure is pretty cool - http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ Someone posted that blog entry to my forums a few days ago. I see several issues with it, mainly cooling and power. Only 6 fans to cool 45 drives? Those are going to bake. The fact that it spikes 14A on power up is going to cause major issues. Real RAID hardware would have delayed start on all those drives. I have delayed start on Areca RAID cards in 6 SuperMicro SuperServers with 8x 250gb SATA drives, and I have to be very careful powering them on otherwise I'll pop my cabinet's breaker :/ I just run a few as cold spares now. If that set up had a real RAID adapter (not fakeRAID), or at least something like ZFS, I'd give it props. Doing a fakeRAID 6 is just asking for failures. I played with fakeRAID on BSD and RHEL for years, before writing both implementations off for production use. I personally wouldn't bother using any RAID at all, but just set them up JBOD. My reasoning being that it is a backup service -- if a drive fails, yes, that backup is gone, but the client should be able to see that the checksum is different and resync. I wanted to build something similar, but with systems instead of drives. I even built a prototype, but power stability was a problem. However, I had /much/ more airflow, and no drives (netboot). Edit: also, anyone remember that whitepaper about SATA drives reaching 2TB in a RAID setting will almost always be guaranteed to fail upon rebuild due to the size of the drive exceeding the drive's own bit error MTBF? optikalus fucked around with this message at 19:11 on Sep 2, 2009 |
|
#
?
Sep 2, 2009 19:08
|
|
|
optikalus posted:Edit: also, anyone remember that whitepaper about SATA drives reaching 2TB in a RAID setting will almost always be guaranteed to fail upon rebuild due to the size of the drive exceeding the drive's own bit error MTBF? Yep. I remember it on a different site, but here's the link I found - http://blogs.zdnet.com/storage/?p=162 Also, remember that google paper a while back, showing little correlation between hdd temperature and failure rate - http://labs.google.com/papers/disk_failures.html. da sponge fucked around with this message at 19:32 on Sep 2, 2009 |
|
#
?
Sep 2, 2009 19:30
|
|
|
$5/user/month for unlimited backup? I wonder how much they count on the average user actually using, to turn a profit? Each pod is $8,000 and that's not counting employees or bandwith/cooling etc.
|
|
#
?
Sep 2, 2009 20:05
|
|
|
optikalus posted:Someone posted that blog entry to my forums a few days ago. I see several issues with it, mainly cooling and power. Only 6 fans to cool 45 drives? Those are going to bake. I think you're wrong here, fan count is a lot less important than CFM forced over the disks. Sure the disks may be hotter than a 2U server with 4 disks in it, but they will be consistently the same temperature. The disks will probably suffer some premature failure, but that is the whole point of RAID. Get a cheapo seagate AS disk with 5 year warranty and just replace them as they fail. Non-delayed startup and initial power surge are certainly very valid problems. If you're doing 14A on boot you can only have ~1 of those per 20A circuit. It makes me wonder if they wander around with an extension cord on a dedicated circuit to turn them on, then migrate it over to the running circuits. Reading the blog it sounds like what they do, turn on the disks, then turn on the cpu/etc Makes me wonder where are they finding a datacenter that can keep themselves even moderately warm for that many watts/sq ft. I imagine with proper ventilation on the hot rows it should be doable, but I can't imagine how many heat exchangers they need. You don't need to keep it really cold, you just have to keep up with the heat put out by the devices and makes sure hot air isn't getting circulated back over to the intake fans in the cold rows.
|
|
#
?
Sep 2, 2009 21:34
|
|
|
It's neat but each power supply runs about half of the components. That's doubling a point a failure and a pretty crappy compromise.
|
|
#
?
Sep 2, 2009 23:10
|
|
|
optikalus posted:Edit: also, anyone remember that whitepaper about SATA drives reaching 2TB in a RAID setting will almost always be guaranteed to fail upon rebuild due to the size of the drive exceeding the drive's own bit error MTBF?
|
|
#
?
Sep 2, 2009 23:46
|
|
|
its absolutely insane for anyone without two very niche aspects to their storage needs: - can be slower than poo poo sliding up hill - can afford a 67TB outage to replace one drive I think I'd trust mogilefs's redundancy policies more than linux's software raid6.
|
|
#
?
Sep 3, 2009 01:05
|
|
|
lilbean posted:It's neat but each power supply runs about half of the components. That's doubling a point a failure and a pretty crappy compromise. Depends on their environment- if they also implement redundancy at the network or application levels (storing the same file on multiples of these boxes) ala waffleimages/lefthand it doesn't really matter if one of the boxes dies on rebuild, just get copies redistributed and bring the node back into the cluster when it's fixed. But at that point you're just reinventing mogilefs/googlefs.
|
|
#
?
Sep 3, 2009 01:15
|
|
|
ragzilla posted:Depends on their environment- if they also implement redundancy at the network or application levels (storing the same file on multiples of these boxes) ala waffleimages/lefthand it doesn't really matter if one of the boxes dies on rebuild, just get copies redistributed and bring the node back into the cluster when it's fixed. But at that point you're just reinventing mogilefs/googlefs.
|
|
#
?
Sep 3, 2009 01:19
|
|
|
|
| # ? Apr 23, 2024 20:55 |
|
|
StabbinHobo posted:its absolutely insane for anyone without two very niche aspects to their storage needs:
|
|
#
?
Sep 3, 2009 01:27
|
|